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The  past  decade  has  witnessed  increased  research  into  the  use  of  linear  statis- 
tical models  to  describe  subjective  human  judgment  processor. . Past  studies  have 
focused  on,  l)  the  superior  predictive  capability  of  statistical  models,  vis-a- 
vis,  man  himself,  2)  the  detection  and  modeling  of  configure I judgment  process, 
and,  3)  the  use  of  judgment  models  as  catalysts  to  conflict  resolution.  This 
research  has  generally  been  conducted  in  the  content  of  clinical  experiments  and 
the  results  have  not  been  extensively  applied  to  practical  decision  situations. 
This  lack  of  diffusion  is  primarily  the  result  of  failure  of  researchers  to 
present  their  models  in  a format  that  the  mm-teclmica  1 derision  maker  can  under- 
s i.niia  and  appreciate. 

This  dissertation  helps  to  bridge  the  gap  between  the  laboratory  and  operational 
environments  by  applying  Policy  Capturing  to  the  decision  process  of  an  install- 
ment loan  officer  who  approves  or  denies  credit  on  the  basis  of  a written  appli- 
cation. The  Automatic  Interaction  Detection  (AID)  algorithm  was  used  to  present 
the  "structural  image"  of  the  policy  in  a format  that  is  readily  comprehensible 
by  the  decision  maker.  A unique  conceptualization  of  configural  decision 
processes  is  proposed  and  demonstrated.  Experimental  resuits  given. 
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ABSTRACT 


The  use  of  linear  statistical  models  to  describe  subjective 
judgment  processes  has  received  considerable  attention  in  the  fields 
of  clinical  psychology  and  psychometrics  over  the  last  decade.  The 
majority  of  this  research  has  been  conducted  within  the  basic  con- 
ceptual framework  of  the  Brunswikian  Lens.  The  studies  have  focused 
on  three  areas.  They  are:  1)  the  superior  predictive  capability  of  the 

statistical  models,  vis-a-vis,  man  himself,  2)  the  detection  and  mo- 
deling of  configural  judgment  processes,  and,  3)  the  use  of  judgment 
models  as  catalysts  to  conflict  resolution.  This  research  has  generally 
been  conducted  in  the  context  of  clinical  experiments  with  only  limited 
implementation  of  the  results  in  the  business,  academic,  and  adminis- 
trative environments.  The  lack  of  diffusion  of  this  innovative  concept 
is  primarily  attributable  to  the  failure  of  the  researchers  to  present 
their  models  in  a format  that  the  non-te chnical  decision  maker  can 

understand  and  appreciate. 

This  dissertation  attempts  to  help  bridge  the  gap  between 
the  laboratory  and  operational  environments  by  applying  the  techniques 
of  Policy  Capturing  to  a practical  decision  process.  The  judgment 
process  used  is  that  of  the  installment  loan  officer  who  approves  or 
denies  credit  solely  on  the  basis  of  information  contained  on  a written 
credit  application  and  the  past  credit  rating  of  the  applicant  The 
Automatic  Interaction  Detection  (AID)  algorithm  is  used  as  the  primary 
means  of  describing  the  loan  officers'  policies,  it  presents  tne 
"structural  image"  of  the  policy  in  a graphical  format  that  is  readily 
comprehensible  by  the  decision  maker.  The  analytical  data  provided 
by  the  technique  is  used  to  hypothesize  the  appropriate  mathematical 
model. 
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A unique  conceptualization  of  configural  decision  processes 
as  a series  of  local  models  over  subspaces  of  the  predictor  set  is  pro- 
posed and  demonstrated.  A heuristic  procedure  for  defining  local 
models  is  presented  and  used  to  develop  policy  models  for  the  loan 
officers.  These  models  achieve  cross-validation  hit  rates  in  excess 
of  91  percent  in  predicting  actual  decisions  on  new  loan  applications. 
Models  for  identifying  the  top  30  percent  of  the  applications,  in  the 
absence  of  credit  rating  data,  are  developed  and  discussed  as  a means 
of  improving  the  level  of  service  that  the  lending  institution  can  provide 
during  non- banking  hours. 
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The  many  definitions  of  Operations  Research  all  involve  a 
common  conceptual  thread;  that  is,  "the  application  of  the  scientific 
approach  and  mathematical  techniques  to  describe,  control,  and  improve 
operations  of  organizational  systems".  This  dissertation  represents  an 
effort  to  adapt,  expand,  and  introduce  yet  another  technique  into  the 
inventory  of  tools  available  to  the  practitioner  of  Operations  Research. 

The  voluminous  literature  in  the  field  of  Operations  Research 
is  replete  with  decision  models  and  solufion  algorithms  for  problems  in 
which  both  the  objective  functions  and  the  relevant  parameters  are 
thought  to  be  well  known.  The  general  statement  of  such  problems  is: 


« $ 
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OPTIMIZE: 

Subject  to: 
equality  constraints: 


~ n 

F (x)  x e V 

h.  (x)  - 0 i = 1 ....  m 
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inequality  constraints: 


giU)  ^ 0 


i = m + 1 ....  p 
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In  all  cases  the  coefficients  of  the  objective  function  and 
constraints  are  either  deterministic  or,  at  least,  estimable  by  some 
probability  distribution.  In  those  cases  where  the  objective  function, 
F(x),  is  not  known,  it  is  either  assumed,  or  the  problem  is  left 
unsolved. 

The  lack  of  objective  functions  has  spurred  research  in  such 
fields  as  "Utility  Theory"  (Fishburn,  1968)  and  "Measurement  Theory" 
(Roberts,  1970),  and  has  brought  about  such  techniques  as  Delphi  (Hel~ 
mer,  1967)  (Dalkey,  1971)  and  the  Echo  Method  (Milburn,  1968). 
However,  these  techniques  have  not  really  been  refined  to  the  stage 
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that  they  represent  readily  usable  tools  for  the  practitioner  of 
Operations  Research. 

The  technique  proposed  here  represents  a more  direct 
attack  on  solving  the  problem  of  determining  what  is  important  to  the 
decision  maker  and  provides  a read  ily  - usable  methodology  for  analyzing, 
describing,  simulating,  and  possibly  improving  the  decision-making 
function.  This  approach  involves  the  inference  of  the  values  of  the 
decision-maker  through  the  analysis  of  his  decisions  over  a sufficiently 
large  number  of  sample  decision  situations.  Development  of  this  type 
of  procedure  uas  advocated  by  Churchman  ( 1 961  ) in  his  philosophical 
discussion  of  optimal  decision  processes. 

The  research  chronicled  by  this  dissertation  includes  the 
exploration  and  adaptation  of  a concept  known  as  "Policy  Capturing" 
for  use  in  ecologically  valid  environments.  The  basic  concept  of  "Policy 
Capturing"  has  been  an  area  of  continued  research  in  the  field  of  clinical 
psychology  during  the  past  decade.  This  research  has  been  pursued  and 
results  reported  under  such  titles  as  "Judgment  Modeling",  "Decision 
Modeling",  and  "Analysis  of  Cognitive  Processes".  Although  there 
has  been  much  work  done  and  many  articles  written  in  the  various 
journals  of  clinical  psychology,  the  transition  from  a clinical  experiment 
to  an  applications  methodology  has  unlv  been  attempted  in  a limited 
number  of  instances.  The  lack  of  diffusion  of  this  innovative  concept 
is  mainly  due  to  the  approach  taken  by  many  researchers  of  working 
with  contrived  and  synthetic  situations,  a practice  which  has  not  en- 
hanced the  adaptability  or  implementability  of  the  scientific  findings 
of  the  psychologist  to  the  extra  - labor atory  or  practical  decision  situ- 
ation. 

It  is  the  purpose  of  this  research  to  broaden  the  applica- 
bility of  the  Policy  Capturing  concept  by  bridging  the  gap  between  the 
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findings  of  th e clinical  psychologists  and  the  realities  of  typical  r>pera- 
tional  decision  situations  facing  the  Operations  Research  practitioner. 
This  bridge  is  afforded  by  the  adaptation  of  recently  developed  satisti- 
cal  techniques  for  the  purpose  of  defining  policy  models  that  reflect 
the  underlying  judgment  process  of  the  decision-maker  and  retain  a 
high  degree  of  predictive  stability. 

This  dissertation  is  divided  into  seven  chapters  and  three 
appendices.  Chapter  1 presents  the  basic  framework  and  the  experi- 
mental paradigm  within  which  the  Policy  Capturing  concept  is  formulated 
and  the  current  research  was  performed.  Chapter  2 provides  a review 
and  analysis  of  the  past  research  that  is  relevant  to  the  current  efforts 
and  Chapter  3 provides  the  focus  and  motivations  for  the  current  re- 
search. Chapter  4 describes  the  de velopnient,  modification,  and  inter- 
pretation of  the  statistical  techniques  used  in  this  research.  This 
chapter  also  presents  a methodology  for  developing  "structural,  image" 
models  that  provide  the  primary  bridge  between  ‘he  concept  of  Policy 
Capturing  and  the  practical  methodology  for  its  application.  Chapter  5 
presents  the  background  and  case  history  of  the  practical  environment 
within  which  this  research  was  conducted  and  the  viability  of  the  method- 
ology is  demonstrated.  Chapter  6 presents  results,  conclusions  and 
implications  generated  by  this  research  for  the  particular  practical 
environment  in  which  it  was  used.  Chapter  7 relates  the  current 
research  to  the  findings  and  hypotheses  generated  by  various  previous 
researchers  and  discusses  the  implications  of  applying  the  basic 
methodology  to  other  environments. 

Finally,  three  appendices  are  provided.  Appendix  A 
reflects  a comparison  of  the  various  statistical  tools  available  and 
their  relationship  with  the  Policy  Capturing  methodology.  Appendix  B 
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provides  the  forms,  data,  and  detailed  structural- image  models  for  the 
case  study.  Appendix  C provides  a self-contained  description  for  using 
the  computer  programs  and  the  analysis  scheme,  developed  in  this 
research,  in  future  Policy  Capturing  efforts. 
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Guide  to  the  Reader: 

Whereas  this  dissertation  attempts  to  straddle  the  gap 
between  the  technical  intricacies  of  statistical  modeling  and  the  opera- 
tional concerns  of  manager,  it  necessarily  addresses  itself  to  two 
audiences.  A guide  to  those  sections  of  interest  to  each  seems  in  order. 
The  reader  who  is  interested  in  the  details  of  HOW  the  analysis  was 
performed  and  the  models  were  built  should  concern  himself  with  Chap- 
ters 1-4,  7,  and  Appendices  A and  C,  The  reader  who  is  interested 
mainly  in  the  results  of  this  application  of  Policy  Capturing  to  modeling 
the  policies  of  loan  officers  will  be  most  interested  in  Chapters  1,  5-7, 
and  Appendix  B. 
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CHAPTER  I:  CONCEPT  OVERVIEW 


The  Judgment  Modeling  Concept: 

The  judgment  process  is  defined  as  "the  process  of  forming 
an  opinion  or  evaluation  by  discerning  and  comparing".  Implicit  in 
this  definition  is  the  existence  of  one  or  more  attributes  or  character- 
istics of  the  situation  that  need  to  be  compared  or  evaluated,  and  the 
identification  of  these  attributes.  If  one  makes  many  judgments  of  the 
same  nature,  the  logical  premise  of  consistency  would  dictate  that  the 
same  set  of  evaluations  and  comparisons  should  be  carried  out  in  each 
decision  situation.  Such  a set  of  comparisons  could  be  classified  as  a 
model  or  policy  for  making  all  judgments  of  a particular  nature.  If 
the  attributes  can  be  quantified  in  some  mathematical  form,  the  com- 
parisons and  evaluations  can  be  carried  out  mathematically  with  the 
policy  being  embodied  in  the  vai  ious  mathematical  operations  to  be 
performed. 

The  fundamental  premise  of  the  Judgment  Modeling  Concept 
is  that  it  is  possible  to  represent  subjective  human  judgment  with  ob- 
jective mathematical  models.  This  premise,  in  itself,  has  been  a 
source  of  considerable  debate  relative  to  the  superiority  of  clinical  vs. 
actuarial  decisions.  This  debate  was  initiated  when  Meehl  (1954)  pro- 
posed that  psychologists  could  make  better  diagnoses,  on  the  average, 
if  they  were  to  actuarily  combine  the  same  data  that  they  normally  used 
in  their  clinical  diagnosis.  The  ensuing  research  has  resulted  in  over 
600  studies  in  the  rather  narrow  field  of  human  information  processing 
in  the  last  decade.  (Slovic  and  Lictensteiri,  1970) 
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The  Judgment  Modeling  Process: 


The  definition  of  judgment  modeling  itself  implies  a rather 
straight-forward,  logical  sequence  of  events  that  has  been  the  basis 
for  most  of  thr  research  in  the  field.  The  functional  elements  of  this 
process  arc  shown  in  Figure  1-1.  As  might  be  expected,  each  element 
has  been  the  focal  point  for  much  detailed  research,  with  particular 
attention  being  paid  to  the  formulation  of  the  mathematical  models  in 
step  number  three  and  to  the  comparison  of  judgment  models  in  step 
number  four.  Very  little  research  has  been  accomplished  on  the  process 
as  a whole  and  virtually  none  has  been  attempted  on  the  process  in 
practical,  extra- laboratory  environments. 

In  an  excellent  review  of  past  research,  Slovic  and  Licten- 
stem  (1  970),  have  reviewed  and  compared  the  Regression  and  the 
Bayesian  Approaches  of  judgment  modeling.  The  basic  differences 
between  these  two  approaches  being  that  in  the  former,  the  judgment 
process  is  formulated  in  the  context  of  the  general  linear  hypothesis 
while  in  the  latter,  the  judgment  process  is  formulated  in  the  context 
of  conditional  probabilities  and  Bayes'  Theorem. 

The  work  pursued  in  the  research  discussed  herein  has 
all  been  wdthin  the  Regression  Approach  and,  therefore,  the  Bayesian 
Approach  will  not  be  addressed.  Further,  within  the  Regression 
Approach,  the  main  thrust  of  this  research  has  been  within  the  "corre- 
lational paradigm"  with  some  efforts  encroaching  on  the  area  tradi- 
tionally known  as  the  "functional  measurement  paradigm".  The  dif- 
ference between  these  paradigms  rests  essentially  in  the  elements 
of  the  judgment  process  considered,  the  statistical  techniques  used, 
and  the  goals  pursued  within  a particular  research  effort. 


FUNCTIONAL  ELEMENTS  OF  TIIE  JUDGMENT  MODELING  PROCESS 


Identify  Candidate  Decision  Situation 

0 


Identify  Attributes  of  Decision 


Identify  Data  Sample  and  Quantify 
2 Attributes  of  Decision 


Determine  Structural  Relationship  and  Propel 
[3  Weighting  of  Attributes 


Compare  Models  of 
Different  Judges 


Formulate  New 
Theories  of 
Decision  Process 


Cluster  Models 
to  get 
Composite 


Use  Models  in 
Actual 
Environment 


Evaluate  Models 
Relative  to  their 
Effectiveness 


FIGURE  l-l 
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The  Brunswikian  Lens: 

The  conceptual  model  for  the  Regression  Approach  of 
judgment  modeling  was  first  proposed  by  Egon  Brunswik  in  1952.  The 
model  as  shown  in  Figure  1-2,  or  minor  modifications  thereof,  has 
served  as  the  conceptual  framework  for  both  the  correlational  and 
functional  measurement  paradigms. 

The  Lens  model  graphically  depicts  the  underlying  hypo- 
thesis that  each  attribute  (cue)  of  the  decision  situation  is  probabilis- 
tically related  to  an  ecological  criterion  of  a "true  state -of-nature" . 
This  relationship  can  be  characterized  by  a general  linear  model  of 
the  form: 


Y = V + P - ? B.  X.  + 

e e e i i,  e i e 


w he  r e . 


the  actual  criterion  value 
the  predicted  criterion  value 

the  error  in  the  prediction 

, ...  . th 

the  coefficient  of  the  l variable 

, , , , .th  . , , 

tne  value  of  the  l variable 

the  subscript  representing  the  ecological 
side  of  the  L ens 


On  the  judgmental  side  of  the  Lead,  a similar  model  is 
hypothesized  which  relates  the  judged  value  of  the  criterion  to  the 


MfcfcH 
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stimulus  cue  values.  This  equation  is: 


Y-  + e- 
3 J 


= x b.  .x.  + 

i i»J  i 


where: 


4* 

ac: 


m 


the  judged  value  of  the  criterion 

the  predicted  value  of  the  criterion  per  the 

judgment  model 

the  error  in  the  prediction 

the  coefficient  of  the  i^  variable 
th 

the  value  of  the  l variable 

the  subscript  representing  the  judgmental  side 
of  the  Lens 


Further  statistics  that  are  relevant  to  using  and  discussing 
the  model  are: 


X. 


R. 


i.  e 


i.  J 


ecological  validity:  correlation  of  the  i^  cue  to 


the  true  criterion  value  (Y  ) 

e 


.th 


utilization  coefficient:  correlation  of  the  i 

cue  to  the  judged  criterion  value  (Y.) 

r^  y : multiple  correlation  between  the  actual 
e e 

and  predicted  criterion  value  in  nature 

ry  y " response  consistency:  multiple  correlation 

j j 

between  decision  makers  judgments  and  the  value 
of  those  judgments  predicted  by  the  model 


■* 


: achievement  index 


r 

a 


r 

m 


r Y Y. 
e J 

rY  Y : matc^^n8  index 
e j 


In  essence,  the  left-hand  side  of  the  lens  represents  a 
modeling  of  reality  and  the  right-hand  side  of  the  lens  represents  a 
modeling  of  the  judge's  perception  and  interpretation  of  reality.  On 
the  ecological  side  of  the  lens,  the  error,  is  made  up  of  random 

error  and  possibly  any  lack-of-fit  of  the  hypothesized  model  to  reality, 
whereas,  on  the  judgmental  side,  e.,  is  composed  of  lack-of-fit  of 
judge's  model  and  his  own  inconsistency  in  following  his  model.  It 
has  been  pointed  out  by  Ward  (1970)  that  the  lack-cf-fit  portion  of  the 
error  in  the  judgmental  equation  can  be  attributed  to  two  main  causes, 

1)  missing  stimulus  variables  in  the  hypothesized  relationship,  and 

2)  incorrect  structural  form  of  the  hypothesized  variables  in  the  model. 


Correlational  versus  Functional  Measurement  Paradigm: 


The  distinction  between  the  correlational  and  the  functional 
measurement  paradigms  can  be  clarified  in  terms  of  the  lens  model. 

In  the  correlational  paradigm,  the  emphasis  is  on  the  judgmental  side 
of  the  lens  with  the  structure  of  the  judgmental  model,  the  cue  coef- 
ficients (b.  .),  and  the  response  consistency  (r.)  being  of  most  interest. 

J J 

Elements  3,  4,  5,  and  8 of  the  judgment  modeling  process  have  been  the 

main  areas  of  concern  for  researchers  within  this  paradigm.  In  the 
functional  measurement  paradigm,  interest  is  still  evidenced  in  the 
judgmental  side  of  the  lens,  but  particular  emphasis  is  put  on  the 
modification  of  the  judgmental  equation  as  a result  of  feedback  from 
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% 


success  and  failures  of  applying  the  model.  Statistics  of  particular 

interest  to  workers  within  this  paradigm  are  the  achievement  index 

(r  ) and  the  matching  index  (r  ),  with  most  of  the  experimental  work 
a m 

occurring  within  elements  1,  2,  3,  6,  and  7 of  the  judgment  modeling 
process. 


Definition  of  Policy  Capturing: 


It  should  be  noted  here  that  in  past  research,  the  term 
"Policy  Capturing"  has  been  generally  used  to  describe  the  analysis 
of  the  judgmental  side  of  the  Brunswikian  Lens.  However,  for  want 
of  more  descriptive  terminology,  the  connotation  of  "Policy  Capturing" 
has  been  expanded  within  this  research  to  be  as  follows: 

IDENTIFICATION  AND  QUANTIFICATION  OF  THE  AT- 
TRIBUTES THAT  ARE  PERTINENT  TO  A DECISION  AND  THE  SUB- 
SEQUENT MATHEMATICAL  DESCRIPTION  OF  THE  DECISION  POLICY 
FOR  THE  EVALUATION  OF  THESE  ATTRIBUTES  is  defined  as 
POLICY  CAPTURING. 

Definitions: 


JS 


At  this  point,  it  is  beneficial  to  explicitly  define  various 
terminology  that  will  be  used  throughout  the  description  of  past  and 
current  research  in  order  to  avoid  any  semantic  difficulties  that 
might  arise  relative  to  various  types  of  mathematical  models. 

Linear  Models:  A linear  model  is  of  the  form 

Y = Y + E = 2 b.  X.  + E i = 1 n 

x 
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A 

where:  Y: 

b.: 

l 

X.: 

i 


E: 


E b,  X.  - predicted  value  of  the  criterion 
3 1 

1 

are  parameters  to  be  estimated 


are  variables  that  are  known  for  any  given  data 

entity  or  that  can  be  compounded  from  other  basic 

3 

variables  such  as  X.  - Z.*W  or  X - Z 

i l i ii 

the  error 


The  definition  "linear"  refers  to  the  fact  that  the  parameters 
"b."  are  constant  over  the  entire  predictor  space  (Vn)  and  does  not 
relate  to  the  order  or  power  of  the  variables  "X.".  All  models  consi- 
dered as  being  linear  models. 

Main  Effects  Models:  A linear  model  in  which  the  highest 
power  of  any  variable  is  unity. 

Curvilinear  Models:  A linear  model  in  which  the  power  of 

some  variable  is  greater  than  unity,  i.  e„  , X^  . 

l 

Configural  Effects  Models:  A linear  model  in  which  cross- 

product  terms  of  the  form  b,.X.X.  where  X.  and  X.  are  different 

iJ  i J i J 

variables,  are  included.  This  is  also  referred  to  as  an  "interaction" 
model. 


Local  Models:  This  is  a linear  model  that  applies  over 

some  suhspace  of  the  predictors,  Sm«  Vn,  and  can  be  considered  as  a 

special  case  of  the  configural  model.  Whereas  in  configural  models, 

the  terms  b.  X.X.  are  generally  considered  to  apply  throughout  the 
” 1 ^ n 

entire  predictor  space,  V , in  local  models,  such  terms  arc  applied 
only  over  the  local  subspace  Sm  . 

Judges:  This  is  the  common  term  for  the  individual  rater 

or  decision-maker  who  is  making  the  judgment.  Many  of  the  past 
works  use  the  terminology  "subjects". 


WgggjjjjttjjjMB 


14 


Cues:  Variables,  attributes,  predictors,  or  other  charac- 

teristics of  the  decision  situation  which  are  perceived  by  the  judge  and 
are  hypothesized  as  the  stimuli  for  his  decisions. 

Historical  Data:  Cue  and  decision  data  taken  from  recorded 
files,  that  is,  the  cues  were  considered  and  the  decisions  made  prior 
to  the  introduction  of  the  concept  of  Policy  Capturing  to  the  subjects. 


CHAPTER  II:  REVIEW  AND  ANALYSIS  OF  PAST  RESEARCH 


Over  view: 


JS». 

' 


* 


The  bulk  of  past  research  in  the  area  of  judgment  modeling 
has  been  accomplished  at  three  institutions,  each  with  a somewhat  sepa- 
rate focal  point.  At  the  Oregon  Research  Institute  (ORI)  the  interest  of 
such  researchers  as  Hoffman,  Slovic,  Goldberg,  and  Dawes  has  been  on 
elements  3,4,  and  5 of  the  judgment  modeling  process  as  shown  in 
Figure  1-1.  Their  main  thrust  has  been  in  evaluating  how  well  a model- 
of-man  can  do  versus  man  himself  and  in  investigating  the  best  way  to 
formulate  the  judgment  models. 

At  Colorado's  Institute  for  Behavioral  Science,  the  focus  of 
researchers  Hammond,  Hursch,  Todd,  Earle,  and  Brehmer  has  been 
on  elements  4 and  7 of  the  process.  Their  main  efforts  have  been  in 
investigation  of  policy  conflicts,  interpersonal  learning,  and  the  training 
of  subjects  to  use  various  types  of  cues. 

The  only  real  experience  in  using  judgment  models  in  a 
practical  environment  to  date  has  been  at  the  Air  Force  Human  Resources 
Laboratory  (AFHRL)  where  researchers  Ward,  Christal,  Koplyay, 
Madden,  and  Mead  have  focused  mainly  on  elements  4,  5,  6 and  7 of  the 
process.  They  have  specialized  in  capturing  and  clustering  the  policies 
that  individual  board  members  use  in  making  USAF  personnel  decisions. 

Although  these  institutions  have  taken  a major  lead  in  the 
past  research  in  Policy  Capturing,  independent  researchers  including 
Einhorn  (elements  3 and  8),  Green  (element  3),  and  Kleinmuntz  (element 
1)  have  made  substantial  contributions  to  the  current  body  of  knowledge 
in  the  field. 
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The  Early  Regrossionists: 

In  I960,  Hoffman  reported  that  one  could  "pararnor phically " 
represent  the  policy  of  a decision  maker  very  well  with  first-order  linear 
regression  models.  He  applied  the  term  "pararnor phic"  to  denote  the 
fact  that,  even  though  the  individual  terms  in  the  policy  model  did  not 
necessarily  represent  meaningful  physical  constructs  relative  to  the 
judge's  logical  processes,  these  terms  could  be  combined  so  as  to  pre- 
dict a set  of  judgments  (usually  rankings)  that  had  a high  correlation 
with  the  actual  judgments.  Hoffman  contended  that  even  though  these 
policy  models  were  paramorphic  and  might  not  portray  the  decision 
process  accurately  from  a structural  standpoint,  they  could  account  for 
a significant  amount  of  the  variability  in  individual  judgments.  He  felt 
that  useful  information  could  be  gained  about  the  judge's  dependence 
on  each  cue  by  examination  of  the  Relative  Weight  for  each  cue.  He 
defined  the  Relative  Weight  for  cue  "i"  in  the  policy  of  judge  "j"  as: 


RW.  . 


6.  . r.  . 

i.J  i.J 


where  : 


standardized  regression  weight  of  the  i cue 

th 

for  the  j judge's  policy. 

correlation  coefficient  of  the  i1^  cue  with  the 

criterion  for  the  j**1  judge.  (Hoffman  calls  this 

the  utilization  coefficient.  ) 

, .th  . 

response  consistency  of  the  j judge. 
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This  contention  was  quickly  refuted  by  Ward  (1962)  in  a demonstration 
that  the  Relative  Weight  was  meaningless  except  in  the  very  special 
case  of  orthogonality  among  the  predictors. 

Consequently,  many  researchers  resorted  to  near  total 
reliance  on  factorially  designed  experiments  in  order  to  have  meaningful 
measure?  of  comparison  among  predictor  variables.  To  facilitate  the 
comparison  of  policies  on  an  inter-judge  basis,  a trend  toward  rather 
strict  control  of  the  data  sets  used  in  the  analysis  also  developed. 
Hoffman  (I960)  justified  such  control  by  stating: 

. . . restricting  the  situation  (by  controlling  the 
stimuli)  assures  that  each  person  is  evaluated 
with  respect  to  the  same  information.  Ambiguous 
and  equivocal  cues  are  removed,  and  all  judges 
are  thereby  certain  to  have  at  their  disposal  the 
same  information  and  no  more.  The  inferences 
made  beyond  this  point  are  certain  to  have  their 
origins  in  the  data  provided,  (p.  118). 


With  his  original  work,  Hoffman  sparked  two  areas  of 
controversy  that  are  germain  to  the  problem  of  applying  Policy  Capturing 
to  an  actual  decision  making  environment.  Hie  first  area  is  the  use  of 
first-order  linear  models  as  an  adequate  representation  of  the  judgment 
process  regardless  of  the  underlying  logical  procedure.  This  approach 
can  present  a nearly  insurmountable  obstacle  to  practical  implementation 
in  convincing  the  decision  maker,  who  may  be  unsophisticated  mathemati- 
cally, that  an  unexplainable  equation  can  do  as  well  as  he  can  with  his 
explainable  judgment  procedure.  The  second  area  is  the  use  of  highly 
structured  and  controlled  data  sets  for  capturing  the  policy  of  the 
decision  maker.  This  technique  often  leads  to  a lack  of  credibility 
since  the  data  used  to  capture  the  policy  may  not  conform  to  the  "real 
world"  data  that  the  decision  maker  must  assess  in  his  nor mal  judgment 


process. 


The  adequacy  of  a paramorphic  first-order  model  must 
really  be  evaluated  relative  to  its  intended  use.  If  prediction  of  the 
final  outcome  is  in  the  major  purpose  and  there  is  no  "selling  to  the 
management"  problem  involved,  then  a highly  efficient  first-order  para- 
morphic is  probably  better  than  a more  complex  configural  model. 

This  is  exactly  the  situation  confronting  the  research  personnel 
at  the  USAF  Human  Resources  Laboratory,  I^ackland  AFB,  Texas.  These 
researchers  were  particularly  concerned  with  providing  decision  aids 
to  the  numerous  personnel  boards  which  make  such  decisions  as  "Who 
should  be  promoted?"  or  "In  which  occupational  specialty  should  a 
recruit  with  a given  set  of  background  qualifications  be  trained?  " The 
goal  in  these  instances  is  adequate  prediction  of  the  composite  decisions 
of  many  judges  for  many  cases  and  not  the  detailed  analysis  of  how  the 
individual  judges  make  up  their  ni’nds.  The  "selling  job"  is  also  mini- 
mized in  that  they  are  operating  in  an  environment  where  they  do  net 
have  to  convince  a given  judge  that  the  policy  model  is  congruent  with 
"the  way  he  thinks  he  makes  decisions",  rather,  the  paramorphic  policy 
model  sells  itself  if  it  is  predictivcly  efficient  in  matching  the  composite 
decisions  of  the  board  members  and  can  result  in  increased  operational 
efficiency. 

The  AFIIRL  has  published  numerous  studies  in  the  develop- 
ment of  policy  models  for  specific  areas.  A few  of  these  areas  include 
assignment  of  personnel  to  jobs  (Ward,  1 962),  merited  pay  and  merited 
grade  for  jobs  (Madden,  1963),  promotion  of  enlisted  men  (Koplyay, 

1969),  and  evaluation  of  job  difficulty  (Mead,  1970).  In  most  of  these 
studies,  the  key  points  of  interest  has  been  the  comparison  of  the  indi- 
vidual policies  of  the  board  members  and  the  attainment  of  a composite 
policy  model.  Thus,  they  have  emphasized  the  clustering  and  compari- 
son of  the  policy  equations  and  have  developed  and  applied  the  JAN 
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Technique.  With  JAN,  the  captured  policies  of  the  individual  judges  are 
hierarchically  clustered  in  a manner  that  minimizes  the  loss  in  predic- 
tive efficiency  of  the  composite  policies  at  each  step.  (Ward  and  Hook, 

1 961  )(Bottenberg  and  Christal,  196l)(Christal,  1963) 

An  example  of  the  information  to  be  gained  about  the  consis- 
tency of  a given  rater  in  applying  "his  policy"  and  about  the  homogeneity 
of  "policies"  within  a group  of  raters  can  be  illustrated  by  considering 

the  results  published  by  Madden.  (1963)  The  data  on  Table  shows 

2 

the  distribution  of  "rater  efficiencies"  (the  multiple  r for  the  rater's 
paramorphic  policy  model)  for  each  of  38  raters  performing  the  ratings 
on  each  of  50  separate  job-descriptor  profiles.  The  "rater  efficiency" 
reflects  the  ratio  of  the  variation  in  ratings  explained  by  the  raters' 
policy  model  to  the  total  variation  in  the  ratings.  It  has  a theoretical 
upper  limit  of  unity.  This  ratio  is  an  indication  of  the  quality  of  the 
"fit"  of  the  policy  model  to  the  policy  used  by  the  rater  in  making  the 
ratings.  Note  that  an  efficiency  of  . 99  was  attained  in  modeling  the 
policy  of  one  individual  relative  to  merited  pay  and  this  efficiency  was 
attained  for  five  individuals  relative  to  merited  grade. 

Table  1.  shows  the  results  of  hierarchial  clustering  of 
the  policy  models  of  the  raters.  At  each  stage  of  the  hierarchial 
clustering  procedure,  the  number  of  groups  or  clusters  is  reduced  by 

one  by  combining  that  set  of  policy  models  that  reduces  the  composite 

2 2 

r the  least  amount.  A sudden  drop  in  r . at  a given  stage  would  indicate 

j J 

that  two  non- homogeneous  clusters  had  been  grouped  at  that  stage.  Note 
that  in  the  last  step  of  the  procedure  for  the  merited  pay  policies,  a 

2 

group  of  37  raters  was  combined  with  a lone  rater  and  the  resultant  r . 
dropped  from  .87  to  .78.  By  using  clustering  techniques,  one  can 
determine  if  significant  differences  in  policy  exist  among  individuals 
or  groups  of  individuals  on  a board.  Then  action  can  be  taken  to  eliminate 


Table  1-1 

RATER  CONSISTENCY 


ft 


Distribution  of  r 's  Achieved  in  Predicting  Rank  Order 

of  Merited  Pay  and  Merited  Grade 
from  Factor  (predictor)  Values 


2 
r . 

J 


Merited 

Pay 


Mer  ited 
Grade 


. 99  1 

. 98  2 

. 97  6 

. 96  6 

. 95  6 

. 94  3 

. 93  1 

. 92  4 

. 91  1 

. 90  3 

. 89  1 

ft  .88  1 

. 87  1 

. 86  1 

. 85 

. 74  1 

. 72 
. 31 
. 15 


5 

7 

2 

2 

2 

2 

0 

3 

0 

2 

5 

1 

2 

1 

1 

1 

1 

1 


N= 
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From:  (Madden,  1963) 
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POLICY  COMPARISON 


Effects  on  r . of  Number  of  Groups 


Basis  of 

Number  of 

Number  of  Judges 

Comp 

Ranking 

Groups 

Each  Group 

r2 

Pay 

38 

1 

. 93 

3 

33,  4.  1 

. 88 

2 

37,  1 

. 87 

1 

38 

. 78 

Grade 

34 

1 

. 92 

3 

16,  16,  2 

. 88 

2 

32,  2 

. 86 

1 

34 

. 84 

Pay  & Grade 

72 

1 

. 93 

3 

64,  5,  3 

. 86 

2 

67,  5 

. 83 

1 

72 

. 81 

50  Subjects:  10  Predictors:  No  Lnteractions:  10  Levels  per  predictor 

*Note  that  only  34  raters  were  grouped  on  the  Merited  Grade  equation 
due  to  obvious  inconsistency  in  4 of  the  raters. 


From:  (Madden,  1963) 
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members  with  deviate  policies  or  to  resolve  conflicts  and  differences 
of  opinion  among  the  raters. 

In  a AFHRL  sponsored  study  of  the  effectiveness  of  identi- 
fying and  resolving  policy  differences  by  first  capturing  the  policies  and 
then  using  the  JAN  Technique,  Mullins  and  Usdin  (1970)  compared  this 
procedure  with  one  in  which  each  judge  estimated  the  relative  impor- 
tance of  each  variable  and  then  the  judges  mediated  their  weights  for 
each  of  the  variables  and  came  up  with  a composite  equation.  They 
found  that  by  using  the  JAN  Technique,  the  quality  of  the  group  judgment 
always  improved  but  that  by  using  the  latter  technique,  the  mediated 
weight  for  a variable  was  often  only  a reflection  of  the  opinion  of  the 
most  powerful,  or  the  most  vocal,  judge.  Further,  they  found  the 
quality  of  the  group  estimate  was  worse  as  often  as  it  was  better. 

The  use  of  policy  aggregation  as  a catalyst  to  conflict 
resolution  was  further  pursued  and  reported  on  by  Stephenson  and  Ward 
(1  970)  in  an  experiment  wherein  they  captured  the  policy  model  of  each 
of  several  members  of  a promotion  board  and  clustered  their  individual 
policy  models.  This  resulted  in  a focusing  of  attention  on  the  previously 
non-isolateable  reasons  for  differences  between  board  members  in  the 
ratings  of  potential  promotees.  With  areas  of  difference  brought  into 
focus  by  means  of  different  weights  on  the  relevant  variables,  some 
limited  progress  was  reported  in  getting  the  board  members  to  come 
to  a common  agreement  on  the  relative  importance  of  the  various 
factors.  They  reported  that  even  though  compromise  on  a policy  did 
not  result  to  the  desired  degree,  the  judges  in  the  study  reported  that 
the  technique  had  served  a very  valuable  purpose  in  clarifying  the 
positions  and  policies  of  each  of  the  board  members. 

Both  of  these  studies  would  suggest  that  policy  capturing 
could  provide  a solution  to  the  problem  in  face-to-face  communications 
described  by  Robinson  (1971)  as  "noise";  that  is,  discussion  of  various 
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interests  and  not  problem  solving. 

In  most  of  the  work  carried  on  or  sponsored  by  the  AFHRL., 
a difference  in  policy  between  several  judges,  or  groups  of  judges,  has 
not  been  decomposed  further  than  necessary  to  effect  a resolution  to  a 
common  policy  or  to  identify  and  eliminate  judges  with  greatly  deviant 
policies.  More  recently,  Brehmer  (1970)  has  pointed  out  tl  it  apparent 
policy  disagreements  should  really  be  decomposed  into  spearate  compo- 
nents, one  part  due  to  individual  policy  inconsistency,  and  the  other  part 
due  to  inter-judge  policy  conflicts.  His  experimental  results  indicate 
that  as  judges  with  policy  disagreements  work  to  reduce  the  conflict 
that  exists  on  an  inter-judge  basis,  there  is  a counter-balancing  effect 
with  reduction  in  policy  conflicts  being  accompanied  by  increases  in 
individual  policy  inconsistency,  resulting  in  very  little  net  change  in 
apparent  differences.  This  would  indicate  that  once  people  recognize 
the  innate  causes  of  policy  conflicts,  they  may  be  willing  to  change 
their  weightings  of  the  cues,  but  they  have  a difficult  time  effecting 
the  change  because  they  cannot  apply  the  new  or  "compromise"  policy 
consistently.  Brehrner  also  found  that  policy  differences  were  reduced 
faster  by  free  communication  and  free  communication  plus  objective 
cue  weights  than  they  were  by  just  communication  of  differences  in 
judgments,  a result  that  would  support  the  utility  of  Policy  Capturing 
as  a catalyst  for  reducing  policy  differences. 

The  Configurality  Debate: 

A somewhat  different  set  of  interests  has  motivated  the 
researchers  for  whom  the  first-order  paramorphic  model  was  untenable. 
They  have  persisted  in  what  Goldberg  (1968)  has  referred  to  as  the 
"search  for  the  configural  judge".  This  continued  effort  to  model  the 
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policy  of  the  judge  with  terms  that  are  more  reflective  of  the  actual 
logic  process  of  the  judge  is  stimulated  by:  1)  the  judges’  introspective, 

beliefs  that  they  process  cues  in  complex  and  configural  ways,  2)  the 
possibility  that  truly  configural  judgment  tasks  have  not  yet  been 
studied,  and,  3)  the  possibility  that  the  experimental  designs  and  sta- 
tistical techniques  available  to  the  researchers  were  best  suited  for 
discovery  of  first-order  effects  and  not  particularly  suited  for  unmasking 
configural  r elationships  . (Slovic  and  Lictenstein,  1970)  Many  publica- 
tions have  appeared  on  both  sides  of  the  "first-order"  versus  "higher- 
order  and  configural"  issue;  the  principle  participants  being  Wiggins 
and  Hoffman  (1968),  Goldberg  (1968),  Hoffman,  Slovic  and  Rorer  (1968), 
Einhorn  (1970,  1971)  and  Valenzi  (1970). 

Evidence  for  the  tremendous  power  of  the  first-order  model 

•* 

was  presented  by  Yntema  and  Torgerson  (1961)  when  they  demonstrated 
that  94%  of  the  variance  within  a completely  configural  design  could  be 
accounted  for  by  a first-order  model. 

Their  analysis  was  based  on  synthetically  generated  data 
from  a purely  configural  3^  design  of  the  form: 

W = aXZ  + bXY  + cYZ 

with  their  paramorphic  surrogate  taking  the  form: 

W = aX  + bY  + c Z 

In  reviewing  the  many  studies  using  first-order  model  up  to 
1970,  Slovic  and  Lictenstein  (1970)  state: 


In  all  of  these  situations  the  linear  model  has 
done  a fairly  good  job  of  predicting  judgments,  as 
indicated  by  r.  values  in  the  . 80' s and  . 90' s for 
the  artificial  ^ tasks  and  the  . 79' s for  the  more 
complex  real-world  situations,  (p.  36) 

On  the  pro- configural  side  of  the  issue,  continued  efforts 

have  been  made  to  find  situations  where  the  higher-order  and  configural 

models  will  consistently  out-perform  the  approximation  afforded  by  the 

linear  surrogate,  superior  performance  being  indicated  by  the  amount 

of  variance  explainable  by  the  policy  model.  In  one  study,  Wiggins  and 

Hoffman  (1968)  compared  the  performance  of  a quadratic  model  and  a 

configural  "sign  model"  to  the  first-order  models  for  29  psychologists 

in  making  a diagnosis  from  mental  symptoms  of  861  cases  of  MMPI 

2 

(Minnesota  Multi-Phasic  Inventory)  data.  The  results  showed  that 
for  16  of  the  29  judges,  configural  models  were  marginally  better,  but 
not  to  a statistically  significant  extent.  In  general,  other  such  efforts 
have  not  been  impressively  successful  in  increasing  the  amount  of 
explained  variance  and  have  caused  some  frustration  to  the  researchers. 
This  frustration  has  led  the  researchers  to  search  for  different  statisti- 
cal methods  and  devise  new  tactics  in  applying  old  tools.  Among  the 
methods  attempted  have  been  the  application  of  Analysis-of- Variance 
procedures  (Hoffman,  Slovic  and  Rorer,  1968)  (Slovic,  1969)  and  the 
"a  priori  partialling  out"  of  variance  due  to  the  hypothesized  configural 
offecca  (Valenzi,  1 970).  Even  with  these  techniques,  statistically  signi- 
ficant evidence  of  configurality  could  not  be  established  by  the  researcher 

Much  of  the  current  literature  is  rife  with  the  misuse  of  the  term 
"linear"  to  denote  "first-order"  models  as  opposed  to  including  all 
models  with  linear  parameters. 

^ This  set  of  data  has  become  known  as  the  "Meehl  data"  and  has  been 
used  extensively  by  various  researchers. 


26 


lending  credence  to  Green's  (1968)  critique  of  all  such  efforts: 

Another  difficulty  with  standard  configural  techniques 
analysis-of- variance,  and  the  new  index  is  that  they 
essentially  are  fishing  expeditions.  The  experimenter 
will  covet  any  configural  effect,  any  interaction  term 
he  can  find.  He  cannot  begin  to  examine  all  the  possible 
non-linear  effects,  and  is  very  likely  to  miss  those 
that  are  present,  unless  he  knows  where  to  fish.  (p.  94) 


Undaunted,  other  researchers  have  taken  the  approach  of 
formulating  completely  new  types  of  models  and  working  with  different 
decision  situations  in  order  to  demonstrate  conf igurality . The  most 
prominent  of  these  efforts  has  been  the  work  by  Einhorn  (1970,  1971) 
in  the  formulation  of  the  "conjunctive"  and  "disjunctive"  models. 
Einhorn,  hypothesized  that  the  appropriate  decision  model  is  really  a 
function  of  the  type  of  decision  task  and  other  factors  of  the  decision 
situation  including:  inter-judge  differences,  distribution  properties  of 

the  cue  values,  number  of  stimulus  objects,  the  correlation  among  the 
cues,  the  number  of  stimulus  or  cue  dimensions  to  be  evaluated  for 
each  object,  and  the  context  within  which  the  decision  is  made.  He 
stated  that  for  certain  types  of  situations,  such  as  those  in  which  the 
strongest  attribute  or  the  weakest  attribute  of  the  stimulus  object  was 
important,  non-compensatory  models  of  either  the  conjunctive  or  dis- 
junctive type  would  apply  respectively.  The  proposed  conjunctive 
model  is  of  the  form: 


N 

y = n 
i=l 


N 

log  Y = 2 a.  log  x. 

i=l  1 


The  index  referred  to  here  is  an  index  of  cue  consistency  proposed  by 
Hoffman  (1968)  as  the  "belie vability  index"  md  relates  to  the  degree 
to  which  separate  cues  match  the  preconceived  ideas  of  how  the 
sep.  rate  cues  should  be  correlated  with  the  criterion. 
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and  the  disjunctive  model  is  of  the  form: 

N b.  N 

Y = n ( )x  or  log  Y = £ -b.  log(a.-x.) 

. , a.-x.  1 1 i 

i=l  l i l-l 

In  his  experiments,  Einhorn  pitted  these  models  against  the 
first-order  models  for  four  judges  in  the  task  of  ranking  graduate  school 
applicants  based  on  Graduate  Record  Exam  scores.  He  found  his  models 
gave  significantly  better  results  for  three  of  the  four  judges  and  the 
shrinkage  of  his  models  upon  cross-validation  was  much  less  than  that 
achieved  with  first-order  models  for  all  four  of  the  judges.  This  last 
finding  is  of  special  interest,  in  that  Ward  (1954)  found  that  configural 
models  generally  did  not  cross  validate  as  well  as  first-order  models 
due  to  "over-fitting".  Hence,  high  cross  validities  for  a configural 
model  would  appear  to  enhance  its  claim  to  authenticity. 

Einhorn' s work  has  offered  new  evidence  for  the  long 
fostered  opinion  that  configurality  existed,  if  it  could  just  be  uncovered. 
It  is  important  to  note  that  Einhorn  himself  recognized  that  his  models 
would  probably  not  perform  as  well  in  different  decision  situations. 

This  admission  did  not  placate  Goldberg,  however,  who  was  apparently 
disgruntled  by  Einhorn's  refutation  of  his  previous  assertion  that  first- 
order  models  fit  best  because  they  are  mathematically  simple  and  man 
thinks  in  simple  ways,  (Goldberg,  1968)  Instead,  Goldberg  (1971) 
immediately  set  about  the  task  of  disproving  Einhorn's  findings  on  the 
much  analyzed  "Meehl  data".  The  results  of  this  vendetta  show-ed  that, 
as  was  nearly  pre-ordained,  the  linear  model  did  a better  job  than 
either  the  conjunctive  or  disjunctive  models  in  depicting  the  judgment 
of  clinical  psychologist  in  making  diagnoses.  This  merely  tends  to 
confirm  Einhorn's  hypothesis  that  the  best  model  is  a function  of  the 
decision  situation.  More  importantly,  in  justifying  his  results, 
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Goldberg  gave  the  first  indication  that  he  and  other  researchers,  within 
the  heretofore  "pure  policy  capturing"  camp,  were  awakening  to  the 
necessity  for  more  emphasis  on  "representative"  research  designs  as 
espoused  by  Brunswik  (1956)  would  be  necessary. 


Bootstrapping: 


Independent  of  the  researchers  belief  or  disbelief  in  HOW? 
the  policy  of  a decision  maker  can  best  be  modeled,  there  is  always 
the  problem  of  justifying  WHY?  they  should  attempt  to  model  the 
decision  maker  at  all.  This  debate  has  continued  since  Meehl's 
original  proposal  of  the  "actuarial  combination  of  cues",  with  some 
clinicians,  apparently  justifying  their  jobs,  claiming  that  the  actual 
combination  of  the  cues  is  only  a small  part  of  the  overall  judgment 
process.  (Holt,  1970) 

Most  of  the  arguments  for  using  policy  models  of  the  judge, 
as  opposed  to  the  judge  himself,  center  around  the  consistency  of  a 
known  and  fixed  computational  formula  as  compared  to  the  inherent 
variability  of  the  human  judgment  process.  One  of  the  main  proponents 
of  using  Policy  Capturing  has  been  Goldberg  (1  970)  who  described  the 
human  reliability  problem  by  saying: 

He  'has  his  days':  Boredom,  fatigue,  illness, 

situational  and  interpersonal  distractions  all 
pla  gue  him,  with  the  result  that  his  repeated 
judgments  of  the  exact  same  stimulus  configu- 
ration are  not  identical.  He  is  subject  to  all 
these  human  frailties  which  lower  the  reliabil- 
ity of  his  judgments  below  unity.  And,  if  the 
judge's  reliability  is  less  than  unity,  there 
must  be  error  in  his  judgments--error  which 
can  serve  no  other  purpose  than  to  attenuate  his 
accuracy.  If  we  could.  . . eliminate  the  random 
error  in  his  judgments,  we  should  thereby  in- 
crease the  validity  of  the  resulting  predictions,  (p.423) 
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In  an  analysis  of  the  performance  of  first-order  policy 
models  versus  the  actual  judgments  on  the  "Meehl  data",  Goldberg 
(1970)  found  that  the  models  were  marginally  better  for  all  29  judges 
than  the  judges  themselves. 

Dawes  (1971)  also  demonstrated  the  superiority  of  policy 
models  over  actual  judgments  in  predicting  the  rank  of  graduate  admis- 
sions. Dawes  and  his  fellow  researchers  at  ORI  have  termed  the 
phenomena  of  the  policy  model's  superiority  to  the  human  in  predicting 
judgments  as  "bootstrapping.  " The  essence  of  the  justification  for 
"bootstrapping"  is  that  mathematical  models,  by  their  very  nature, 
are  abstractions  of  the  processes  that  they  model.  The  efore,  if  the 
decision  maker's  behavior  involves  following  valid  principles,  but  he  is 
unable  to  follow  them  reliably,  these  valid  principles  will  be  abstracted 
by  the  model,  and  once  abstracted,  the  superior  reliability  of  the  machine 
can  be  enlisted  to  follow  them  exactly,  resulting  in  bettor  quality  judg- 
ments. It  is  recognized  by  this  argument  that  this  reduction  in  variabi- 
lity will  occur  if  there  is  no  systematic  deviation  from  the  principles 
based  on  the  variables  themselves,  i.  e.  , if  the  structural  form  of  the 
model  is  correct.  On  this  point,  Dawes  asserts  that  bootstrapping  can 
occur  even  if  there  are  structural  defects  of  the  model,  such  as  in  cases 
where  a first-order  model  is  used  to  represent  a truly  configurai  deci- 
sion situation,  if  the  cues  themselves  are  monotonic  and  the  data  is 
fallible. 

Dawes  and  Diller  (1970)  derived,  and  experimentally 
verified,  a mathematical  criterion  based  on  the  lens  model  parameters 
for  determining  when  bootstrapping  could  be  expected  to  occur  for  the 
case  of  first-order  models  of  very  good  (near  - optimal)  judges.  This 
condition  for  bootstrapping  is: 
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the  predictability  of  the  judge  (r^  . q ) 

j J 

reliability  of  the  residuals  (Y. -■$.),  i.  e.  , 

J J 

the  correlation  of  residuals  in  cross- 
validation  samples 

reliability  nf  criterion  values  in  cross- 
validation  samples 
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reliability  of  predicted  correlation  values 
(Y.)  in  cross-validation  samples. 


In  this  same  study,  they  proposed  amalgamating  the  first- 
order  prediction  of  the  judge  from  tiie  policy  model  with  the  judge's 
actual  prediction  to  got  a hybrid  prediction.  This  amalgamation  is 
based  on  weighting  the  current  judgment  and  the  policy- model  predicted 
judgment  in  the  form: 
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Their  experimental  work  on  this  procedure  indicated  that 

the  ama lgamation  of  predicted  and  judge  criterion  values  was  superior 

to  the  judge  alone  or  the  prediction  alone,  especially  in  the  case  where 
4 

non-linear  cue  use  uas  considerable. 

Bowman  (1%3)  and  Kunreuther  (1  969),  have  also  proposed 
and  studied  the  uro  of  policy  models  of  export  managers  to  reduce 
variance  in  management  decisions  relative  to  production  management 
and  inventory  control.  They  have  suggested  that  such  a technique 
would  be  an  excellent  surrogate  for  the  more  traditional  analysis,  much 
of  which  is  dependent  on  relationships  of  decision  and  responses  that 
are  greatly  separated  in  time,  or  which  is  based  on  cost  parameters 
that  are  intangible  and  must  be  estimated  or  assumed.  This  suggestion 
is  tantamount  to  stating  that  it  is  more  efficient  to  model  the  expert's 
opinion  of  the  proper  policy  to  follow,  than  to  model  the  actual  response 
as  a function  of  the  decision  policy.  This  rather  intriguing  thought  of 
using  experts  to  estimate  the  cue-response  relationships  when  the 
actual  relationships  were  unattainable  was  also  studied  by  Mullins  and 
Usdin  (1970).  In  that  study,  the  use  of  judges  to  estimate  the  relation- 


^Here  "non-linear"  refers  to  "higher  order  and  configural"  per  the 
definition  in  Chapter  I. 
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ships  between  the  actual  criterion  values  anfl  the  cues  produced  esti- 
mated criterion  values  that  were  virtually  no  different  than  the  actual 
criterion  values. 

Sawyer  (1966)  analyzed  the  results  of  some  45  studies 
accomplished  prior  to  1966  and  proposed  that  the  judgment  task  be 
divided  into  two  subtasks,  measurement  and  prediction.  He  concluded 
that  the  perceptive  capabilities  of  the  human  were  best  put  to  use  in  the 
measurement  task  and  that  the  inherent  reliability  of  the  machine  made 
the  policy  models  superior  in  the  prediction  task.  Sawyer's  lead  in 
decomposing  the  global  judgment  task  into  these  two  subtasks  has  more 
recently  been  followed  by  Einhorn  (1972).  Einhorn  has  agreed  with 
Saw’yer  and  proposes  that  not  only  should  the  measurement  task  be 
relegated  to  the  expert  judge,  but  that  the  whole  process  might  be 
improved  even  further  by  using  multiple  judges  in  the  measurement 
task  with  each  judge  being  an  expert  in  the  measurement  of  some 
particular  attribute.  Such  a technique  would  then  use  the  expert 
measurements  as  an  input  to  an  actuarial  combination  of  the  cues  in 
attaining  a global  judgment. 

Apart  from  the  justification  of  the  relative  accuracy  and 
efficiency  of  using  policy  models,  the  WHY?  of  Policy  Capturing  can 
be  answered  relative  to  management's  effective  use  of  computers  in 
the  current  man-machine  era.  Many  authors  such  as  Yntema  and 
Torgerson  (1961)  have  discussed  management's  need  to  "teach  the 
computer  how  they  want  decisions  to  be  made",  and  others  state  that 
in  the  operation  of  large  organizations  in  the  future,  "specialist's 
knowledge  will  be  stored  in  computer  banks".  (Vandell,  1970)  Since 
computers  are  best  at  manipulating  mathematical  expressions,  the 
use  of  captured  policy  models  to  instill  man's  value  judgments  in  the 
computer  appears  to  be  the  only  logical  approach  to  the  task. 
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The  Nature  of  the  Decision  Process: 


In  addition  to  studying  the  more  mundane  aspects  of  policy 
modeling  such  as  the  practicality  of  using  these  models  as  decision 
aids,  a very  considerable  effort  has  been  undertaken  by  psychologists 
in  using  Fbticy  Capturing  as  a descriptive  tool  for  understanding  the 
fundamental  cognitive  processes  of  the  decision  maker.  Some  of  these 
resuits  have  been  the  direct  goal  of  specific  research,  while  others 
have  come  about  as  by-products  of  the  more  applied  research  efforts. 

In  any  event,  it  is  well  to  be  cognizant  of  a number  of  these  findings  in 
any  project  in  which  a viable  set  of  policy  models  is  to  be  sought  for  use 
in  a practical  environment. 

One  of  the  original  postulates  in  the  area  of  cognitive 
processes  was  the  concept  of  "cognitive  strain"  advanced  by  Bruner, 
Coodnow,  and  Austin.  (1956)  They  hypothesized: 

Where  the  nature  of  a task  imposes  a high  degree 
of  strain  on  the  memory  and  inference,  the  strategy 
used  for  coping  with  the  task  will  tend  to  be  less  con- 
ducive to  cognitive  strain.  To  put  it  in  terms  of  an 
analogy,  if  someone  has  to  move  a heavy  weight,  there 
is.  . .likelihood  that  the  mover  will  have  recourse  to 
strain  reducing  techniques  for  carrying  out  his  task. 

(P-  112) 


Another  of  the  fundamental  postulates  resulted  from  the 
work  of  Miller  (1956)  who  found  that  there  are  rather  low  limits  on 
the  number  of  items  of  data  that  the  human  can  evaluate  and  integrate 
in  a consistent  manner. 

In  line  with  these  postulates,  Einhorn  (1971)  found  that  the 
judge  made  more  consistent  judgment  when  fewer  cues  were  available. 
He  proposed  that  man  uses  excess  data  in  some  hybrid  fashion  instead 


of  discarding  ii  as  might  be  expected  in  any  effort  to  reduce  cognitive 
strain.  Similarly,  Slovic  and  Lietenstein  (1970)  concluded  that  the 
results  of  past  research  showed  that  more  information  gave  the  judge 
a false  confidence  without  improving  his  prediction  consistency. 

Slovic  (1969)  investigated  the  relative  uses  of  first-order 
and  higher-order  cues  in  individual  and  group  judgments.  He  found 
some  evidence  that  when  multiple  cues  were  present,  the  tendency  to 
use  curvilinear  models  was  less  in  the  case  of  group  judgments,  while 
it  was  more  in  the  case  of  individual  judgments.  This  suggests  that 
the  averaging  effect  of  group  judgments  may  tend  to  obscure  configur ality 
in  individual  policies,  especially  if  the  corifigurality  is  different  for 
different  judges. 

Earle  (1970)  studied  the  learning  anu  inter  per  sonal  learning 
characteristics  of  judges  trained  on  first-order  and  higher-order  cues. 
He  found  that  those  judges  trained  initially  to  use  a higher  order  cue 
could  readily  adapt  to  using  a first-order  cue  without  the  aid  of  a 
teacher,  but  that  those  judges  who  had  been  trained  on  a first-order 
cue  had  to  receive  help  in  adapting  to  the  use  of  a higher-order  cue. 

Another  point  of  investigation  has  been  in  the  area  of 
sequential  decision  processes.  Although  most  of  this  work  has  been 
carried  out  within  the  previously  mentioned  Bayesian  Approach,  Slovic 
(1966)  and  Hoffman  (1968)  have  proposed  that  the  judge  first  assesses 
the  "belie veability"  of  the  cues  and  then  proceeds  to  adjust  the  cue 
weights  of  his  first-order  policy  model  accordingly.  In  a more 
exhaustive  work,  Bettman  (1969)  essentially  pursues  this  same  notion 
except  that  he  hypothesizes  that  the  decision  maker  first  assesses  the 
consistency  of  the  cues  and  given  they  are  consistent,  uses  a simple 
or  first-order  model;  however,  if  they  are  inconsistent,  the  decision 
maker  uses  a configural  model  in  what  Bettman  calls  a "problem 
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solving"  approach.  Those  findings  are  suggestive  of  the  "lexicographic" 
model  mentioned  by  Einhorn  (1970)  in  which  it  is  hypothesized  that  a 
decision  is  made  on  the  most  important  variable  first  and  then  remaining 
variables  are  considered  in  the  order  of  their  importance.  Unfortunately, 
Einhorn  and  others  defer  from  pursuing  this  model  due  to  their  inability 
to  mathematically  model  the  process. 

Summary: 


Probably  the  strongest  general  impression  to  be  gained  from 
a review  of  the  past  research  efforts  in  Policy  Capturing  is  the  artificial 
nature  of  the  envir onments  in  which  the  research  has  generally  been 
conducted.  This  is  especially  paradoxical  in  that  one  of  the  basic  tenents 
of  the  Brunswikian  Lens  conceptualization  is  the  use  of  "representative" 
designs.  Only  recently,  researchers  such  as  Goldberg  (1  970),  Einhorn 
(1  972),  and  Dawes  (1971)  have  begun  to  voice  concern  over  this  departure 
from  reality,  with  Dawes  expressing  his  reservations  of  past  research 
results  in  the  statement: 

Most  of  these  studies  have  been  conducted  in  analogue 
experimental  situations  and  it  is  reasonable  to  ask 
whether  the  situations  were  constructed  in  such  a way 
that  the  linear  model  works,  but  are  not  representa- 
tive of  the  real  world  decision  making  situations.  (p»  181) 

Most  of  the  past  analyses  have  been  performed  on  decision  tasks 
where  only  a limited  number  of  well-defined  cues  have  been  used.  As 
a typical  example,  Valenzi  (1970)  arbitrarily  reduced  the  number  of 
cues  in  his  study  from  20  to  5 based  on  a preliminary  interview  with 
one  subject  in  the  study.  Such  practices  would  seem  to  be  risky  in  the 
face  of  data  indicating  that  even  experienced  judges  have  a difficult 
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time  accurately  describing  the  relative  weights  of  the  variables  in  their 
policies  (Slovic  and  Lictenstein,  1970).  Valenzi' s own  results  indicated 
that  only  one  of  the  four  judges  in  his  study  was  able  to  correctly  estimate 
the  relative  importance  on  the  variables  in  their  policies. 

In  most  studies  even  the  procedure  of  asking  the  subject 
which  variables  they  "felt"  were  important,  before  deciding  on  a pre- 
dictor set,  has  been  omitted  with  the  subjects  being  given  a set  of  pre- 
defined cues.  This  practice,  which  is  equivalent  to  ignoring  steps  1 
and  2 of  the  Policy  Capturing  process  as  depicted  in  Figure  ] -1,  page  7. 
was  first  challenged  by  Garner  (1  970)  when  he  noted  that  the  psychologists 
modeling  clinical  judgments  had  paid  virtually  no  attention  to  the  problems 
of  identifying,  scaling,  and  coding  appropriate  stimuli.  Einhorn  (1971) 
later  asserted  that: 

The  practice  of  presenting  cues  to  the  judge  in  a 
decomposed  form  not  only  imposes  the  experimenter 1 s 
own  judgment  as  to  what  the  relevant  cues  actually 
are  but  more  importantly  it  does  a considerable  part 
of  the  cognitive  work  for  the  judge,  (p.  8) 


A related  problem  with  the  artificial  environment  is  the 
extent  to  which  the  experimenters  have  relied  upon  the  statistical  and 
computational  niceties  of  factorial  designs  in  choosing  their  stimulus 
data  sets.  In  an  early  study  of  the  possible  effects  of  using  sample 
data  sets  that  were  statistically  different  than  the  underlying  population, 
Dudyeha  and  Naylor  (1966)  perturbed  the  predictor  inter  cor  relation 


matrix  from  the  true  values  and  studied  the  resultant  effects  on  the 
policies  derived  for  the  judges.  They  found  that  perturbation  of  the 
interr elationships  between  predictor  variables  to  minimize  the  correla- 
tions resulted  in  lower  rater  consistency  and  better  discrimination  of 
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the  effects  of  the  individual  variables,  while  increasing  these  correlations 
resulted  in  higher  rater  consister  ;y  and  worse  discrimination  at  the  indi- 
vidual variable  level.  This  would  indicate  that  artificial  orthogonalization 
of  the  predictor  variables  by  use  of  factorial  designs  with  equal-cell 
frequencies,  would  tend  to  produce  policies  that  were  less  predictive 
than  the’  should  be.  Hence,  computational  efficiency  has  been  purchased 
with  decreases  in  accuracy  in  such  experiments. 

In  addition  to  these  rather  arbitrary  decisions  relative  to 
which  variables  and  which  data  units  to  use  in  the  experiment,  the 
criterion  for  evaluation  of  the  various  policy  models  has  usually  been 

restricted  to  using  the  percentage  of  explained  variance,  as  expressed 

2 

by  the  squared  multiple  correlation  coefficient  (r^  ).  Slovic  and 

Lictenstein  (1970)  found  that  most  of  the  models  that  they  reviewed 

were  NOT  cross- validated.  Further  review  by  this  investigator, 

indicates  that  only  in  the  case  of  those  models  developed  or  sponsored 

by  the  researchers  at  the  AFHRL,  and  in  a few  other  cases  such  as 

Bettman  (1969),  has  cross-validation  of  the  derived  policy  equations 

been  accomplished  on  a systernmatic  basis.  Admittedly,  explained 

variance  is  a commonly  used  "measure  of  goodness",  but  with  small 

2 

data  samples  models  can  achieve  inflated  values  of  r^  solely  due  to 
the  idiosyncratic  nature  of  the  sample  data. 

A second  general  impression  of  the  past  work  is  the  rather 
restricted  range  of  modeling  possibilities  that  have  been  entertained. 
Within  the  correlational  paradigm  and  the  Brunswikian  lens  framework, 
almost  ail  of  the  work  has  been  confined  to  linear  first-order,  higher- 
order,  and  configural  models  of  the  type: 


£ b.x.  + ...  + b.x.  + b. . x.  x.  + E 
- i 1 i J 1 I 
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where  the  values  of  h.,  b.,  and  b..  are  considered  to  apply  over  the  total 
n 1 J 

predictor  space,  V . Hoffman  (1968)  characterized  such  models  as 
being  n-dimensional  hyperplanes  and  states: 


The  concept  of  a best-fitting  hyperplane  implies  sort 
of  an  averaging  of  the  effects  of  all  points  in  the 
multiple  dimensioned  space,  and,  since  linearity  is 
assumed,  the  regression  coefficients  are  invariant 
over  the  entire  stimulus  space,  (p.  59) 


Even  Einhorn's  (1970)  departure  from  the  more  standard  main-effects 
or  main-effects-plus- configural-effects  modeling  approaches  by  using 
the  conjunctive  and  disjunctive  models  still  represents  the  fitting  the 
entire  predictor  space  by  a single,  continuous  hyperplane. 

The  only  works  that  reflect  a departure  from  this  concept 
are  the  efforts  of  Slovic  (1968),  Hoffman  (1968)  and  especially  Bcttman 
(1969).  In  the  first  two  of  these  studies,  it  is  proposed  that  the  judge 
adjusts  the  coefficients  in  his  first-order  model  based  on  an  initial 
assessment  of  cue- consistency  or  "believab’lity",  while  in  the  latter 
study,  the  whole  structure  of  the  policy  model  is  dependent  upon  this 
initial  assessment.  These  procedures  effectively  divide  the  predictor 
space  into  a set  of  discontinuous  hyperplanes,  each  hyperplane  repre- 
senting a local  policy  or  a local  model.  Bettman  (1969)  appears  to  be 
the  only  one  of  these  researchers  to  recognize  this  implication  with 
his  conclusion: 


The  major  and  most  useful  conclusion  to  draw  from 
the  above  . . . (research).  . . is  that  a linear  or  less 
configural  model  can  be  viewed  as  a complex  model 
conditioned  by  a given  cue  configuration,  (p.  134) 
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The  real  potential  of  the  concept  of  modeling  the  entire  pre- 
dictor space,  V , with  a series  of  disjoint  hyperplanes  over  subspaces, 
Sm.  of  the  predictors  has  not  been  further  addressed.  However,  this 
technique  presents  a possible  solution  to  the  problem  of  obtaining  models 
that  retain  the  desirable  characteristics  of  first-order  models  while 
accounting  for  the  configural  logic  processes  that  many  judges  feel  they 
use.  One  possible  reason  that  this  area  has  remained  unexplored  is  the 
problem  of  determining  the  basis  upon  which  to  deaggregate  the  total 
predictor  space  into  appropriate  subspaces.  Such  a division  would 
surely  have  to  be  based  on  the  inherent  characteristics  of  the  policy 
and  the  predictors  for  it  to  be  meaningful.  Any  arbitrary  approach 
would  be  analogous  to  Green's  (1968)  "fishing  expeditions"  and  would 
be  doomed  to  frustration.  Blaylock  (1  969)  voices  a similar  opinion 
in  his  discussion  of  building  theories  with  multivariate  mode  Is  by 
stating: 


One  cannot  simply  build  theories  involving  interaction 
effects  without  some  rationale  to  suggest  the  conditions 
under  which  non-additive  relationships  might  be  expected, 
(p.  156) 


Thus,  a viable  methodology  for  identifying  appropriate 
subspaces  over  which  local  models  could  be  fit  would  appear  to  be  a 
necessary  precursor  to  the  exploration  of  the  approach  of  using  multi- 
ple hyperplanes,  as  opposed  to  using  a single  hyperplane,  in  the 
modeling  of  policies. 

As  a final  comment  in  this  review,  the  observation  of 
Slovic  and  Lictenstein  (1970)  that  resulted  from  their  own  exhaustive 
review  of  the  previous  research  is  germain.  They  concluded  that  for 
as  much  work  as  had  been  accomplished  in  many  different  areas  of 


judgment  modeling,  surprisingly  little  cross-communication  among  the 
efforts  had  transpired.  They  compared  this  situation  to  a speeding 
vehicle  without  side- windows,  the  only  visibility  being  directly  down 
the  path  of  travel  of  a given  paradigm,  or  modeling  strategy-  With 
this  comment  they  have  noted  the  requirement  for  a side- window;  the 
review  of  the  recent  work  of  others  such  as  Hoffman,  Bettman,  and 
Einhorn  suggests  a faint  outline  as  to  where  a side-window  might  be 
carved;  it  is  the  purpose  of  the  work  which  follows  to  open  this  side- 
window  so  that  future  researchers  can  look  out  and  the  practitioners 
of  Operations  Research  can  look  in. 


CHAPTER  III:  FOCUS  OF  THIS  RESEARCH 


Application  of  Policy  Capturing  in  Practical  Environments: 


The  introduction  of  the  Policy  Capturing  Concept  and  a review 
of  the  past  work  can  provide  the  genesis  for  many  tantalizing  ideas  of 
the  potential  benefits  that  such  a technique  could  offer  to  Managers, 
Educators  and  other  Administrators  in  extra-laboratory  environments. 
However,  as  with  any  scientific  concept,  a problem  of  the  conversion 
from  a fairly  general  collection  of  thoughts  to  a specific  methodology 
exists  in  the  adaptation  of  the  Policy  Capturing  Concept. 

The  review  of  past  research  pointed  out  that  such  a conversion 
has  only  been  previously  attempted  in  a few  instances,  specifically,  those 
studies  undertaken  by  the  researchers  associated  with  the  Air  Force 
Human  Resources  Laboratory  (AFHRL),  In  these  cases  the  effective 
implementation  of  the  concept  has  been  materially  aided  by  the  particu- 
lar decision  tasks  and  the  environments  involved;  namely,  decision 
tasks  in  which  the  predictors  were  such  quantities  as  test  scores, 
aptitude  indices,  and  performance  ratings;  data  sets  that  consisted  of 
large  quantities  of  data;  underlying  judgment  processes  that  were  very 
well  represented  by  first-order  policy  models;  and,  minimal  emphasis 
upon  correlation  of  the  paramorphic  models  with  the  actual  logical 
procedures  of  particular  judges.  ^ Thus,  even  the  AFHRL  work  has  been 

T ’ . 

In  the  latter  aspect,  the  dillerence  between  a policy  model 
that  is  visible  and  can  be  explained  and  a policy  model  that  is  reflective 
of  a particular  judge's  self- per ceived  logical  process  is  important.  The 
only  requirement  for  the  former  is  that  the  variables  and  structural  re- 
lationships in  the  model  be  relatable  in  some  rational  manner  to  the  de- 
cision process;  the  requirement  in  the  latter  is  that  those  relationships 
must  not  only  be  rational,  but  also,  they  must  be  in  harmony  with  the 
way  the  individual  judge  ''believes"  he  makes  decisions. 
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in  environments  with  attributes  that  make  them  pseudo- clinical  in 
nature  since  they  are  closer  to  ideal  experimental  situations  than  can 
be  generally  expected  in  the  business,  academic,  and  governmental  or 
other  administrative  types  of  environments.  In  the  extra- laboratory 
environments,  categorical  predictors,  limited  data  sets,  complex  and 
configural  logic  processes,  and  skepticism  for  "black  box"  answers 
are  probably  more  characteristic  of  the  decision  situations, 

The  focus  of  this  dissertation  is  directed  at  exploring  the 
hypothesis  that  Policy  Capturing  can  be  applied  effectively  in  the  extra- 
laboratory environment.  The  fundamental  assumption  underlying  this 
hypothesis  is  that  the  successful  application  of  Policy  Capturing  to 
practical  decision  tasks  is  dependent  upon  providing  the  decision 
modeler  with  an  efficient  methodology  for  deriving  explainable  policy 
models  that  reflect  the  mental  procedures  that  the  decision  maker  feels 
he  goes  through  in  making  judgments.  This  assumption  is  the  basis 
for  the  first  part  of  the  effort  of  the  research,  i.  e.  , the  analysis  of 
various  modeling  alternatives  and  the  development  of  a methodology 
that  will  lead  to  models  that  are  both  accurate  and  descriptive  of  the 
underlying  logical  processes.  The  stated  hypothesis  is  the  basis  for 
the  remainder  of  this  research,  i.e.,  the  application  of  Policy  Capturing 
in  an  extra-laboratory  environment  and  the  analysis  and  demonstration 
of  the  benefits  attainable  by  applying  the  Policy  Capturing  technique  in 
this  environment. 

At  this  point,  a further  characterization  of  which  types  of 
decision  tasks  would  probably  be  most  amenable  to  the  use  of  Policy 
Capturing  is  warranted.  In  making  such  a characterization,  one  might 
conceptualize  all  decision  tasks  as  lying  in  a two-dirnensional  envelope 
such  as  is  shown  in  Figure  3-1.  One  axis  of  this  envelope  corresponds 
to  the  inherent  subjectivity  of  the  decision  situation  and  the  other  axis 


CHARACTERIZATION  OF  DECISION  TASKS 
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Factual  Predictor 
Relationships  with 
Possible  Subjective 
Measurement 


Subjective  Predictor 
Relationships  and 
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EXAMPLE  JUDGMENTS 

Computation  of  Perimeter  Length;  only  judgment  involved  is  in 
reading  measurement  scale,  A--perimeter  of  a circle;  B-- 
perirneter  of  a rectangle 

Acceptability  for  Graduate  School  (Dawes,  1971);  two  judgment 
involve  weights  for  variables  and  evaluation  of  the  quality  of 
the  undergraduate  school  (OS). 

Probability  of  Being  Hired  as  Secretary  (Valenzi,  1970);  up  to 
20  different  predictors  could  have  been  used. 

Evaluation  of  Job  Difficulty  (Mead,  1970);  both  subjective  and 
quantitative  variables  used  to  predict  job  difficulty. 


FIGURE  3-1 
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corresponds  to  the  dimensionality  of  the  predictor  space.  Any  decision 
situation  in  which  the  relationship  between  the  predictors  and  the  criterion 
cannot  be  fully  explained  by  some  immutable  natural  law  would  fall  some- 
where within  this  envelope.  Any  decision  situation  within  the  envelope 
has  a requirement  for  some  degree  of  human  judgment  in  making  the 
final  decision.  At  one  end  of  the  subjectivity  axis,  the  selection  of  which 
equation  or  physical  law  should  be  applied  is  the  only  part  of  the  total 
decision  left  to  the  judge,  while  on  the  other  end  of  this  axis,  the  judge 
is  totally  responsible  for  the  subjective  identification  of  the  relevant 
variables,  the  assessment  of  their  individual  relationships  with  the 
criterion,  and  the  combination  of  their  contributions  into  a global 
judgment. 

Policy  Capturing  would  be  particularly  applicable  in  the 
region  of  this  envelope  where  considerable  fuzziness  exists  in  the 
relationship  between  predictors  and  the  criterion,  and  in  that  region 
where  the  dimensionality  of  the  problem  is  high. 

The  more  mechanical  aspects  of  decision  scenarios  that 
solicit  the  application  of  the  technique  are: 

1)  The  decision  task  is  routine  and  repeated  on  a fre- 
quent basis  with  the  relative  importance  of  the  various 
predictors  remaining  stable  in  the  short  term. 

2)  There  are  multiple  decision  makers  for  whom  a 
composite  decision  is  desired  or  whose  individual 
decisions  should  be  consistent. 

3)  The  quantity  of  the  data  for  each  decision  task  might 
lead  the  judge  to  simplify  the  decision  process  and 
focus  on  different  subsets  of  the  cues  from  case  to 
case  in  an  effort  to  reduce  cognitive  strain. 
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4)  The  decision  process  is  possibly  subject  to  biasedness 
and  the  detection  and  removal  of  this  biasedness  would 
be  beneficial. 

5)  The  actual  predictor / criterion  relationship  can  only 

be  determined  after  a considerable  time  lapse  and  such 
a time  interval  can  render  any  model  of  this  relation- 
ship inappropriate  and  obsolete  due  to  environmental 
changes. 

6)  The  relative  importance  of  the  parameters  of  the 
decision  process  are  not  known  and  are  not  estimable 
from  explicit  physical  relationships. 


♦ 
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Given  these  basic  characteristics  of "fuzzy"  decision 
processes  and  the  mechanical  aspects  of  target  applications,  a more 
detailed  discussion  of  several  functions  that  Policy  Capturing  serve  is 
possible.  Five  generic  roles  in  which  Policy  Capturing  would  be 
beneficial  are: 

A . Factoring  Expert  Judgment  into  Automated 

Decision  Making 

As  noted  in  the  Introduction,  the  Operations  Research 
practitioner  has  long  been  concerned  with  optimization  of  value  based  on 
models  of  various  operational  systems,  but  these  models  often  include 
imprecise  "guess-timates"  of  Management's  utilities  for  risk  and 
profit  in  the  form  of  objective  function  coefficients.  The  more  precise 
approach  of  implicitly  determining  a manager's  expert  opinion  of  the 
relevant  variables  and  their  weights  via  Policy  Capturing  would  assure 
that  the  essence  of  the  manager's  expertise  is  factored  into  the  decision 
model  and  not  just  those  aspects  of  his  experience  that  are  easy  to 
quantify  or  verbalize.  Whereas  the  objective  functions  of  many  organi- 
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zations  are  really  a reflection  of  the  composite  judgment  of  managers 
at  many  levels,  the  use  of  explicit  policy  models  for  the  contributors 
would  provide  an  improved  basis  for  discussion  and  integration  of  policy 
opinions  of  each  manager  in  a manner  commensurate  with  his  proper 
role  in  the  company.  With  less  explicit  procedures,  a person's  impact 
upon  organizational  policies  are  often  based  solely  upon  his  debating 
ability,  dominant  personality  or  title,  and  not  upon  the  quality  of  his 
opinions. 

B.  Standardization  of  Policy 

The  use  of  Policy  Capturing  techniques  to  assess 
intra-judge  consistency,  inter-judge  consistency,  and  even  inter-organi- 
zational consistency  would  appear  to  have  merit.  On  an  intra-judge 
basis,  the  inability  of  a given  judge  to  make  consistent  judgments  over 
a set  of  similar  judgmental  situations  could  indicate  a fundamental 
uncertainty  in  his  perception  of  the  task  or  possibly  a higher  susceptibility 
to  random  external  pressures.  On  an  inter-judge  basis,  the  assessment 
of  the  consistency  of  two  or  more  persons  performing  identical  decision 
tasks  would  help  to  make  the  performance  of  this  task  invariant  to  which 
judge  makes  the  decision. 

The  use  of  Policy  Capturing  to  detect  systematic 
differences  on  an  inter-organizational  basis  is  particularly  intriguing 
from  the  standpoint  of  administering  a large  de- centralized  organization. 
In  cases  where  a policy  is  set  at  some  central  headquarters,  but  admin- 
istered at  various  subordinates  levels  such  as  divisions  of  large 
corporations  or  regional  offices  of  government  agencies,  Policy  Cap- 
turing could  provide  a vehicle  for  identifying  communications  and 
interpretation  problems  between  the  policy  makers  and  the  policy 
administrators. 
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C.  Training 

The  use  of  Management's  policy  models  would  also  be 
of  value  in  the  training  of  new  personnel.  This  is  especially  true  where 
the  judgment  process  is  highly  subjective  and  Management  is  continually 
faced  with  the  dilemma  of  whether  to  use  the  experts  to  "do-the-job"  or 
to  "train-the-new-people".  Codifying  the  expertise  of  the  seasoned 
employee  in  the  form  of  policy  models  and  using  these  models  as  training 
aids  would  enhance  the  transfer  of  knowledge  and  experience  without  ! 

requiring  continual  participation  of  the  expert  judge.  Such  models  could 
also  be  used  as  a baseline  or  standard  with  which  experienced  employees, 
who  may  have  lost  contact  with  the  lower  level  decision  making  process, 
could  "recalibrate"  themselves. 

D.  Policy  Simulation 

Once  a policy  has  been  captured,  the  modification  o; 
the  policy  model  to  simulate  changes  in  the  policy  could  provide  a valu- 
able tool  for  Management  in  performing  a "sensitivity  analysis"  and 
assessing  impacts  of  such  changes  on  an  a priori  basis.  This  applica- 
tion was  proposed  by  Ward  and  Davis  (1963)  as  an  efficient  method  for 
analyzing  many  possible  policy  alternatives  before  institution  of  any 
changes,  especially  in  cases  where  the  effect  of  a change  might  take  a 
considerable  time  to  become  evident. 

E.  Mechanization  of  Routine  Judgments 

The  phenomena  of  "bootstrapping"  and  the  inherent 
reliability  of  machines  presents  possibly  the  greatest  opportunity  for 
benefit  to  be  gained  from  Policy  Capturing.  Not  only  docs  the  reliability 
of  the  machine,  in  evaluating  and  integrating  numerous  stimulus  vari- 
ables, offer  advantages,  but  further,  the  potential  increases  in  speed  and 
efficiency  to  be  gained  in  performing  the  function  with  minimal  human 
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intervention  are  substantial.  The  use  of  the  machine  allows  for  the 
analysis  of  more  data  than  can  be  used  by  the  human  on  a continual  basis. 
For  example,  in  the  only  previously  published  estimate  of  savings  to  be 
gained  by  using  policy  models,  Dawes  (1971)  estimates  that  approximates 
$18  million  could  be  saved  annually,  on  a national  basis,  by  mechanizing 
the  evaluation  of  graduate  student  applications  using  captured  policies 
ot  admission  boards.  Substantial  savings  can  be  imagined  even  if  the 
computer  does  not  make  the  final  decision,  but  only  pre-processes  the 
data  and  provides  its  evaluation  to  the  judge  in  such  a form  that  allows 
the  judge  to  make  his  decision  more  efficiently. 
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The  Transition  from  Hypothetical  to  Methodological: 

Speculating  and  hypothesizing  as  to  the  potential  of  a concept 
can  generate  considerable  enthusiasm,  but  such  activities  do  not  consti- 
tute verification  that  the  concept  is  anything  other  than  a dream.  If  a 
concept  is  purely  theoretical,  the  researcher  can  set  about  defining 
ppstulates,  lemmas,  and  theorems  which  will  result  in  necessary  and 
sufficient  logical  proofs  of  the  concept's  validity.  However,  if  one  is 
dealing  with  such  "fuzzy"  entities  as  man  and  the  human  judgmental 
process,  he  is  confined  to  the  world  of  empiricism  wherein  the  only 
available  approach  to  verification  is  demonstration.  Given  numerous 
repeated  demonstrations  over  a broad  range  of  conditions,  these  empiri- 
cal demonstrations  can  result  in  acceptance  and  tacit  verification  of 
the  concept.  However,  such  verifications  only  hold  within  the  bounds 
of  the  underlying  assumptions  and  population  characteristics  for  which 
the  demonstrations  were  performed,  a point  often  overlooked  when 
assumptions  become  so  commonly  used  that  they  cease  to  be  empha- 
sized in  the  analysis  and  description  of  the  results. 


Many  researchers  who  find  themselves  operating  in  the 
world  of  empiricism  try  to  regain  some  measure  of  "theoretical  legiti- 
macy", or  at  least  try  to  simplify  the  verification  procedure,  by  relying 
upon  the  theoretical  developments  of  mathematical  statistics  and  demon- 
strating "statistical  significance".  This  too,  requires  the  making  of 
assumptions  which  often  go  unsubstantiated,  being  justified  only  by  the 
claim  that  they  are  commonly  used.  Among  the  most  commonly  used 
assumptions  are  those  of  randomness,  univariate  or  multivariate  nor- 
mality, independence,  and  homoscedacity . The  carte  blanche  use  of 
these  assumptions  is  one  of  the  major  problem  areas  in  translating  many 
of  the  findings  from  the  clinical  environment  to  operational  environments. 
If  one  is  to  effectively  apply  his  results  to  the  real-world  he  must  derive 
these  results  under  real-world  experimental  conditions  or  demonstrate 
that  any  underlying  assumptions  do  not  vitiate  his  results  in  the  real 
world. 

Criteria  for  Attaining  Effective  Models: 

Consistent  with  the  philosophy  of  performing  Policy 
Capturing  with  ecologically  valid  data  and  being  fully  cognizant  of  any 
assumptions,  the  criterion  for  the  measurement  of  the  quality  of  a 
model  must  also  be  congruent  with  the  requirements  in  the  environment 
being  modeled.  Establishment  of  such  requirements  represents  a further 
delineation  of  the  previously  stated  fundamental  assumption  of  this 
research.  The  basic  characteristics  chat  models  should  have  to  be 
viable  in  the  business,  academic  and  administrative  fields  are: 

A)  Accuracy:  The  models  must  possess  at  least  that 
degree  of  accuracy  necessary  to  produce  a "bootstrapping"  effect  and 
to  provide  an  improvement  in  operational  efficiency. 


B)  Reflectivity  and  Interpretability : The  models  should 

be  reflective  of  the  logical  processes  of  the  judge  in  order  that  he  may 
identify  with  the  model  and  have  confidence  in  it. 

C)  Communicability:  The  model  should  be  readily  com- 

municable if  it  is  to  be  used  as  a training  device.  Simplicity  in  a model 
would  add  to  its  communicability  to  and  by  non- scientifically  oriented 
judges. 

D)  Data  Amenability:  The  model  must  be  able  to  accept 

and  use  data  that  is  defined  and  categorized  similarly  to  the  manner  in 
which  the  judge  perceives  and  uses  it  in  his  normal  judgment  procedure. 

E)  Data  Conservatism:  The  modeling  procedure  should 

be  operable  with  data  quantities  that  are  not  prohibitively  large  in  terms 
of  data  collection  and  should  not  require  major  disruption  of  the  on- 
going decision  process  for  data  collection. 

The  above  delineation  of  the  characteristics  of  effective 
models  h'lps  to  clarify  the  criteria  for  measuring  model  goodness,  but 
the  very  nature  of  these  characteristics  does  not  readily  admit  them  to 
quantitative  measurement.  The  researcher  is  still  It' ft  with  subjectivity 
in  the  evaluation  of  various  models;  however,  the  dimensions  of  this 
subjective  judgment  have  been  better  identified. 

These  criteria  are  not  only  qualitative,  but.  their  applica- 
bility is  also  dependent  upon  the  functional  application  that  is  intended 
for  the  model.  Tabic  3-i  reflects  an  attempt  by  this  researcher  to  list 
the  three  most  important  model  characteristics  for  each  of  the  functions 
that  were  identified  previously. 


51 


% 


TABLE  3-1 


FUNCTION 

Factoring  "Expert  Judgment" 
Standardization  of  Policy 
Training 

Policy  Simulation 
Policy  Mechanization 


MODEL  CHARACTERISTICS 

A,  B,  C 

A,  B,  C 

B,  C,  D 
A,  D,  E 
A,  B,  D 


A 
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The  Research  Approach: 

The  foregoing  definition  of  the  experimental  arena  and  the 
research  objectives  is  a necessary  prelude  to  the  definition  of  the 
specific  research  approach  and  research  tasks  to  be  accomplished. 

The  two  major  objectives  of  the  effort  are: 

1)  the  analysis  and  development  of  a modeling  methodo- 
logy that  will  produce  policy  models  with  the  desired 
characteristics,  and, 

2)  the  demonstration  of  the  potential  viability  and  utility 
of  policy  models  of  an  actual  decision  process  in  an 
extra-laboratory  environment. 

This  research  effort  is  unique  with  respect  to  past  judgment 
analysis  studies  since  it  includes  the  pursuit  of  both  of  these  objectives 
simultaneously.  The  effort  to  adapt  and  apply  a new  methodology  for 
defining  policy  models  is  actually  embedded  within  the  effort  to  obtain 
and  validate  policy  models  in  an  operational  environment.  In  light  of 
the  current  state-of-the-art  in  judgment  mode.1ing,  the  success  of  the 
methodology  development  effort  is  considered  a pre-requisite  for  the 
accomplishment  of  the  second  of  these  objectives.  In  terms  of  the 


52 


functional  steps  of  the  traditional  Policy  Capturing  process  as  depicted 
in  Figure  1-1,  page  7 , the  total  research  effort  represents  the  tailoring 
of  all  of  the  functions  to  accommodate  the  conditions  in  an  extra-laboratory 
decision  situation.  More  importantly,  the  methodology  effort  involves  the 
development  of  a completely  new  procedure  for  sLep  #3  of  that  process. 

In  this  respect,  two  separate  contributions  are  expected 
from  this  research.  The  first  is  a set  of  procedures  or  guidelines 
for  constructing  policy  model  in  extra-laboratory  derision  situations. 

These  guidelines  and  the  discussion  of  the  model  building  methodology 
are  addressed  in  detail  in  Chapter  IV.  The  second  contribution  consists 
of  the  facts  and  implications  that  are  generated  in  the  process  of  con- 
ducting the  Policy  Capturing  exercise  in  the  particular  decision  situation. 
The  background  of  the  decision  environment  and  the  results  of  the 
Policy  Capturing  exercise  are  discussed  in  the  format  of  a case  study  in 
Chapter  V and  Chapter  VI. 

An  Overview  of  the  Research  Tasks: 

The  fundamental  task  of  developing  a model  building 
technique  is  accomplished  through  the  definition  of  "structural  image" 
models  with  the  Automatic  Interaction  Detection  (AID)  procedure.  AID 
was  recently  developed  by  a group  of  sociologists  at  the  University  of 
Michigan  for  the  purpose  of  discovering  structural  relationships 
between  variables  in  large  quantities  of  sociological  data. 

The  motivation  for  pursuing  this  approach  is  the  similarity 
between  the  conceptual  diagram  of  a configural- effects  judgment  process, 
as  shown  in  Figure  3-2,  and  the  binary  tree  that  results  from  the  appli- 
cation of  the  AID  technique.  Figure  3-2  schematically  diagrams  the 
configural  effects  judgment  process  in  which  the  relevant  stimulus 
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variables  and  their  relative  importance  is  obviously  dependent  upon  the 
value  of  the  previously  considered  stimulus  variables.  In  a conceptually 
similar  manner,  AID  performs  a "local  analysis"  of  the  data  and  builds 
a binary  tree  that  reflects  the  "structural  image"  of  the  data.  In  this 
tree  each  node  represents  a subgroup  of  the  total  data  set.  Each  sub- 
group is  uniquely  defined  by  a common  set  of  predictors  and,  each  split 
or  branching  reflects  the  most  important  variable  relative  to  reducing 
the  unexplained  variability  in  the  data  at  that  node.  Hence,  the  splitting 
process  reflects  the  importance  of  the  predictor  variables  relative  to 
those  predictor  variables  defining  the  subgroup  or  node.  If  interaction 
and  configural  relationships  exist  among  the  predictors,  it  can  be 
detected  via  analysis  of  the  predictors  involved  in  the  splits  and  the 
relative  importance  of  each  predictor  after  each  split. 

In  this  research,  a conceptual  link  between  the  "structural 
image"  model  of  AID  and  configural  judgments  processes  is  made  in 
terms  of  modeling  judgment  processes  with  a series  of  local  hyper- 
planes. A local  hyperplainj  is  a hyperplane  defined  over  a subspace  of 
the  predictor  set. 

The  major  subtasks  within  this  effort  that  relate  specifically 
to  the  building  and  application  of  policy  models  include: 

1)  Identification  of  an  extra- laboratory  decision  situation 
in  which  most  of  the  potential  benefits  from  policy 
capturing  could  be  derived. 

Z)  Identification  of  all  potentially  pertinent  variables 
based  on  discussion  with  actual  decision  makers. 

3)  Collection  of  data  representative  of  the  population  of 
decision  cases. 

4)  Coding,  analysis,  and  refinement  of  data  categoriza- 
tions and  predictor  definitions. 
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5)  Collection  of  decisions  by  each  participating  judge 
on  codified  predictors  over  the  data  set. 

6)  Investigation  of  alternative  modeling  approaches. 

7)  Development  of  models  for  each  of  the  participants. 

8)  Verification  and  field-testing  of  models  in  an  actual 
extra- laboratory  environment. 

9)  Analysis  of  benefits  and  problems  in  actual  application 
of  the  policy  models. 

The  decision  process  chosen  in  subtask  1 as  the  basic 
vehicle  for  this  research  is  that  of  the  installment  credit  officer  in 
granting  or  denying  installment  credit  on  durable  appliances.  This 
decision  is  based  on  credit  applications  forwarded  to  the  credit 
institution  by  the  appliance  dealer  and  does  not  involve  direct  personal 
contact  between  the  decision  maker  and  the  applicant.  Several  of  the 
more  pertinent  reasons  for  using  this  situation  as  the  basis  for  the 
research  are: 

1)  The  decision  process  is  felt  to  be  configural  in 
nature. 

2)  The  data  and  decisions  are  real  and  typical  of  the 
types  of  data  and  decisions  made  throughout  business, 
academic,  and  governmental  agencies. 

3)  The  decision  task  is  a routine  and  frequent  part  of 
the  judges  daily  duties. 

4)  The  judge  has  an  economic  motive  for  making  the 
proper  decision  in  the  most  efficient  and  accurate 
manner  and  is  willing  to  use  any  decision  aids  that 
are  understandable  and  effective. 

5)  Most  of  the  hypothesized  benefits  of  Policy  Capturing 
could  accrue  in  thi  ; environment. 


i 
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Attainment  of  meaningful  results  in  this  application 
should  constitute  a demonstration  of  the  general 
viability  and  utility  of  the  Policy  Capturing  methodology 
in  the  extra-laboratory  environment. 


Subtasks  2,  3,  4,  and  5 are  accomplished  with  the  objective 
of  minimizing  the  influence  that  the  procedures  and  decisions  of  the 
data  definition,  collection,  or  coding  phases  have  upon  the  derived 


policy  model; 


Particularly  important  aspects  of  these  subtasks  include: 
Initial  identification  and  coding  of  the  predictors  is 
accomplished  through  extensive  discussions  with  the 
loan  officers  involved  with  refinement  and  recoding 
of  the  predictors  variables  resulting  from  the  initial 
analytical  efforts, 

The  decision  data  consists  of  a random  sample  of 
approximately  half  of  the  actual  decision  cases  that 
occurred  during  a five-month  period  from  May  to 
September,  1971.  Actual  decisions  made  prior  to  the 
start  of  the  project  are  used  in  one  part  of  the  analysis. 
Decisions  subsequently  collected  for  each  of  five  loan 
officers  on  the  codified  version  of  the  predictor-  vari- 
ables are  also  used. 


This  emphasis  in  collecting  data  and  decisions  "au  naturel"  represents 
a departure  from  the  usual  procedures  of  policy  modeling  studies  which 
work  with  predefined  predictors  and  fixed  categorizations.  This  approach 
complicates  the  whole  Policy  Capturing  procedure,  but  hopefully  justifies 
itself  by  resulting  in  models  that  are  more  realistic  and  easier  to  imple- 
ment in  actual  environments. 


Subtasks  6 and  7 involve  the  development  of  first-order  and 
second-order  policy  models  for  the  judges  and  the  comparison  of  these 
models  with  the  "local  models"  attained  through  the  application  of  the 
techniques  resulting  from  the  methodological  development  effort, 

Finally,  the  collection  and  use  of  cross-validation  data  by 
the  field  test  approach  in  subtask  8 is  unique  to  this  study.  This  sub- 
task not  only  provided  a measure  of  cross-validation  of  the  derived  models 
but  also,  identified  potential  implementation  problems  and  served  as  a 
demonstration  of  the  viability  of  the  concept  to  the  participants. 

Summary: 


The  focus  of  this  research  project  is  unique  in  that  it  not 
only  involves  the  entire  Policy  Capturing  process,  and  the  application 
of  the  entire  process  in  "real-world"  environments,  but  it  also  involves 
a major  methodology  adaptation  as  an  integral  part  of  research. 

The  primary  goal  of  the  research  is  obtaining  policy  models 
that  are  "irnplementable".  The  criteria  for  "implementable"  models 
are  subjective  at  best,  but  a set  of  desirable  model  characteristics  has 
been  proposed  in  this  chapter.  The  absence  of  these  characteristics  may 
not  dictate  in  viability  for  a model,  however,  the  fundamental  assumption 
of  this  research  is  that  possession  of  these  attributes  will  enhance  the 
viability  of  a model.  As  noted  by  Rosenthal  (1966),  such  an  assumption 
reflects  a bi?.s  in  the  attitude  of  the  investigator  and  could  be  a signifi- 
cant determinant  in  the  results.  Therefore,  the  research  approach  has 
been  formulated  so  as  to  enhance  the  credibility  of  the  results  as  much 
as  possible  by  modeling  actual  decision  makers  who  were  making  actual 
decisions  in  their  natural  environment,  and  then  subjecting  the  resulting 
models  to  the  most  rigorous  of  all  validation  tests  --  analysis  of  their 
performance  under  actual  environmental  conditions. 


CHAPTER  IV:  THE  DEVELOPMENT  AND  USE  OF 
STRUCTURAL  IMAGE  MODELS  IN  POLICY  CAPTURING 


Introduction: 

This  chapter  presents  the  conceptualization,  an  example, 
and  guidelines  for  the  modeling  of  configural  judgment  processes  as 
a series  of  local  models  which  define  hyperplanes  over  subspaces  of 
the  predictor  set,  The  adaptation  and  use  of  the  AID  algorithm  as  a tool 
in  portraying  the  structural  image  of  the  judgment  process  and  in 
defining  appropriate  subspaces  and  hypothesizing  local  models  is  dis- 
cussed. 

The  technique  is  applied  to  define  local  models  for  the 
judgment  data  published  by  Valenzi  (1970)  as  well  as  to  define  the  policies 
of  installment  loan  officers.  The  results  of  the  former  analysis  are 
compared  with  the  results  published  by  Valenzi  for  his  second-order 
regression  and  ANOVA  models.  The  modeling  effort  for  the  loan 
officers  comprises  a major  effort  for  this  research  and  is  discussed  in 
detail  in  Chapters  V and  VI. 

Conceptualization  of  the  Configural  Judgment  Model  as  a Series  of 
Subspacc  Hyperplanes: 

Past  modeling  of  judgment  processes  as  either  first-order, 
higher-order,  or  configural  linear  statistical  models  has  generally  been 
in  the  context  of  "continuous"  models.  In  this  discussion,  the  descriptor 
"continuous"  has  the  following  connotation: 
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1.  If  a model  is  of  first-order,  the  coefficients  of  each 
of  the  predictors  applies  to  all  values  which  that  pre- 
dictor can  assume. 

2.  If  the  model  has  interaction  terms,  the  configural 
relationship  between  the  interacting  variables  is 
assumed  to  be  the  same  over  all  possible  values  which 
the  individual  variables  can  assume  and  the  coefficient 
for  the  configural  term  is  applicable  to  all  possible 
combinations  of  these  values.  Hence  the  adjective 
"continuous"  is  relating  to  the  domain  over  which  the 
terms  of  the  model  are  defined  and  does  not  relate 
solely  to  the  "linearity"  of  the  model  coefficients. 
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Continuity  has  been  inherently  imposed  through  the  formu- 
lation of  the  judgment  models  as  linear  regression  models  of  the  form: 
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where  the  coefficients,  "b.",  apply  over  the  entire  predictors  space, 

vn. 

As  discussed  in  Chapter  II,  Hoffman  (1968)  and  others 
have  described  such  models  as  representing  the  "best  fitting  hyperplane" 
over  the  predictor  set  with  the  coefficients  reflecting  the  "average" 
effects  over  the  entire  stimulus  space. 

The  near-universal  representation  of  judgment  processes 
as  "continuous"  models  appears  to  largely  have  been  a function  of  the 
availability  and  the  under  standing  of  the  standard  regression  computa- 
tional routines  and  the  types  of  predictors  that  were  considered  in  the 
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models.  The  alternative  of  formulating  judgment  models  in  terms  of 
discrete  or  binary  regression  models  has  been  mysteriously  avoided. 

This  avoidance  may  be  due  to  the  fact  that  the  predictors  considered 
in  past  studies  have  generally  been  interval  variables.  In  those  cases 
where  there  were  categorical  predictors,  the  approach  usually  has  been 
to  attempt  various  multidimensional  scaling  techniques  to  transform 
the  variables  into  interval  form. 

There  are  certain  advantages  to  "continuous  formulations,  " 
two  in  particular  being  the  economy  of  degrees  of  freedom  required  by 
the  predictor  vectors,  and  the  parsimony  in  describing  models  that 
have  fewer  predictors. 

On  the  other  hand,  binary  or  categorical  regression  formu- 
lations, in  which  each  possible  value  of  each  predictor  represents  a 
separate  category,  consumes  one  degree  of  freedom  per  category  and 
results  in  one  predictor  vector  and  one  coefficient  for  each  category. 

Such  models  do  not  represent  hyperplanes,  but  rather,  a scries  of 
unique  pointH  in  the  stimulus  space.  The  advantage  of  binary  models 
is  that  they  inherently  avoid  the  assumption  of  continuity  and  each  of 
the  coefficients  reflects  the  exact  effect  for  a single  point  in  the 
stimulus  space.  Thus,  there  is  no  "averaging"  of  the  effects  represented 
by  a particular  coefficient. 

In  the  past  research,  the  use  of  ANOVA  models  by  the 
researchers  at  the  Oregon  Research  Institute  and  by  Valenzi  (1970) 
are  the  only  serious  ventures  into  the  use  of  categorical  models.  * 

^Appendix  A discusses  equivalency  of  ANOVA  and  categorical/binary 
regression  models. 


Clearly,  a compromise  between  the  de  facto  assumption  of 
continuity  and  the  alternative  of  considering  each  point  in  the  predictor 
space  is  desirable  to  reach  an  optimum  tradeoff  between  parsimony  and 
exactness. 

A viable  approach  to  such  a compromise  appears  to  lie  in 
the  definition  of  a mixed  model  in  which  both  the  parsimony  of  the  con- 
tinuous model  and  the  exactness  of  the  discrete  model  are  utilized.  In 
terms  of  hyperplanes  and  subspaces,  this  approach  envisions  defining 
a number  of  discrete  subspaces,  each  of  which  can  be  modeled  ade- 
quately with  its  own  hyperplane. 

In  reference  to  judgment  processes  this  corresponds  to 
defining  subprocesses  over  various  patterns,  or  subsets,  of  the  cue 
variables.  Past  researchers,  Slovic  (1969),  Hoffman  (1968),  and 
Bettman  (1969),  actually  provide  the  genesis  for  this  type  of  conceptu- 
alization with  their  suggestion  that  judges  may  actually  think  in  a se- 
quential fashion.  Their  contention  that  man  first  surveys  the  predictor 
values,  assesses  their  consistency  or  belie vability,  and  then  selects 
and  applies  the  appropriate  decision  logic  is  conceptually  analogous  to 
the  initial  definition  of  a particular  subspace,  followed  by  the  applica- 
tion of  a model  within  that  subspacc. 

In  terms  of  the  "structural  image"  of  a configural  effects 
judgment  process  as  shown  in  Figure  3-2,  page  53  , the  subspaces 
could  be  represented  by  the  branches  of  the  tree  and  the  hyperplanes 
are  defined  by  those  variables  considered  in  any  branch. 

At  the  left  of  the  tree,  one  subspace  is  defined  as  consisting 
of  all  possible  categories  of  the  individual  predictors  and  all  possible 
combinations  thereof.  After  the  fir  st  branching,  two  subspaces  are 
defined  on  the  basis  of  the  predictor  used  in  that  stage.  In  terms  of 
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Hoffman's  (1968)  work,  that  predictor  may  be  some  index  of  "believabi- 
lity";  in  terms  of  Bettman's  (1969)  work,  it  could  be  an  index  of  cue 
"consistency".  Each  sequential  branching  results  in  a further  de-aggre- 
gation of  the  predictor  space,  and  in  the  limit,  a completely  categorical 
model  would  be  defined  in  which  each  subspace  consisted  of  a single 
pattern  of  predictor  values. 

If  one  pursues  this  analogy  between  a sequential  decision 

process  and  the  division  of  the  entire  predictor  space  into  subspaces, 

the  major  problem  in  modeling  the  process  becomes  one  of  defining 

the  appropriate  subspaces.  This  problem  is  combinatorial  in  nature. 

Considering  a decision  process  with  five  trichotomous  predictors, 

5 

there  are  3 (243)  individual  predictor  combinations,  or  single-point 
2 243 

subspaces.  Furthermore,  there  are  2 - 1 (a  very  large  number  ! ) 

possible  combinations  of  these  points  into  unique  patterns  of  subspaces. 
Obviously,  a trial  and  error  search  would  be  prohibitive  and  an  effi- 
cient method  of  defining  the  subspaces  is  required. 


The  Automatic  Interaction  Detection  Algorithm: 
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The  Automatic  Interaction  Detection  (AID)  algorithm  was 
originally  developed  by  J.  A.  Sonquist  and  J.  N.  Morgan  of  the 
Institute  for  Social  Research,  Ann  Arbor,  Michigan.  In  a series  of 
monographs  and  presentations,  they  proposed,  explained,  tested,  and 
expanded  the  computer  code  for  the  algorithm.  (Sonquist  and  Morgan, 


'This  would  be  the  case  if  enough  data  points  existed  such  that  each 
possible  combination  of  the  predictors  existed  in  the  data  set.  Other- 
wise, the  maximum  possible  number  of  combinations  is  the  upper 
limit  on  the  number  of  single  point  subspaces. 


.& 


1 964)(Sonquist,  1967)  {Sonquist,  1 970)(Sonquist,  Baker,  and  Morgan, 
1971) 

The  algorithm  accomplishes  the  sequential  division  of  the 
data  set  into  subsets  based  on  that  split  which  causes  the  greatest  re- 
duction in  the  unexplained  variability.  The  procedure  involves  the 
iterative  application  of  one-way  analysis  of  variance  over  every  possible 
split  in  the  predictor  set.  The  major  logical  steps  of  the  algorithm  are 
depicted  in  Figure  4- 1 ; a detailed  mathematical  description  is  given  in 
Appendix  C. 

The  output  from  this  computer  program  consists  of  a series 
of  splits  and  a set  of  terminal  subgroups  which  best  explain  the  varia- 


bility  in 

the  criterion 

variable.  A model  of  the  form: 
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where: 

b. 

i 

= 

the  mean  response  of  those  data  units  falling 
in  subgroup  "i". 

X, 

l 

a binary  group  membership  flag  for  those  data 
units  falling  in  subgroup  "i". 

can  be  written  for  these  subgroups.  A major  benefit  of  the  program 
is  the  graphical  display  of  the  splitting  process  in  the  form  of  an  AID- 
Trce  as  shown  in  Figure  4-2. 

This  graphic  display  of  the  splits  was  one  of  the  primary 
motivations  for  developing  the  program  since  it  allows  visualization 

3 

The  relationship  between  the  AID  model  and  regression  models  arc 
discussed  in  Appendix  A. 
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LOGIC  OF  AID  ALGORITHM 


Select  subgroup  -with  largest  total  sum  of  squares 

a.  Check  limits  on  minimum  allowable 
group  size. 

b.  Check  limits  on  minimum  TSS  in  group. 


Further  splitting  possible? 


Yes 


1 Perform 

one-way  ANOVA  for  each  possible  split  of 

j each  predictor 

a. 

Order  categories  per  mean  response  if 
they  are  not  monotonic. 

b. 

Perform  (k-1)  one-way  ANOVA  calcula- 
tions for  predictor  with  "k"  categories. 

c. 

Save  best  split  on  each  predictor. 

Select  best  split  on  best  predictor  and  attempt  split. 

a.  If  resulting  groups  too  small,  abort 
split. 

b.  If  resulting  groups  large  enough,  split 
parent  into  two  new  subgroups. 
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of  the  inherent  structure  in  the  data  A second  motivation  was  the 
elimination  of  the  requirement  for  assuming  that  linearity  and  additivity 
were  properties  possessed  by  the  proper  model.  In  essence,  AID  is  a 
heuristic  approach  to  searching  the  raw  data  for  structure  and  is  purely 
a mechanization  of  the  procedure  a researcher  might  go  through  manually 
in  hypothesizing  a model. 

Sonquist  and  Morgan  (1964)  hypothesized  that  the  structure 
of  the  proper  model  would  be  indicated  by  the  variables  used  in  the 
splitting  process  and  by  the  shape  of  the  AID- Tree.  They  further  hypo- 
thesized that  symmetrical  trees  would  be  indicative  of  additive  terms  in 
the  model  which  could  be  properly  modeled  by  first-order  predictors. 

More  importantly,  they  felt  that  asymmetry,  especially  relative  to  the 
predictors  involved  in  the  splitting  process,  would  be  indicative  of 
configural,  or  interactive,  relationships  among  the  variables.  In  this 
regard,  they  categorized  tree  structures  as  being  either  of  the  "trunk- 
twig"  or  of  the  "trunk-branch"  type.  The  trunk-twig  structure  could 
be  either  of  the  top  termination  or  of  the  bottom  termination  types. 

In  the  top  termination  type,  small  homogeneous  subgroups  (twigs) 
would  be  split  off  from  the  tree  with  each  subgroup  possessing  some 
particular  advantage.  In  the  bottom  terminating  tree,  the  twig  sub- 
groups would  contain  data  units  which  possessed  alternative  disadvantages. 
In  the  trunk-branch  structure,  the  tree  would  be  reasonably  symmetric 
and  would  be  reflective  of  cumulative  advar  ages  and  disadvantages. 

Sonquist  (1970)  presented  the  results  of  his  continuing 
evaluation  of  the  AID  computer  program  as  his  doctoral  dissertation. 

In  Lhis  evaluation  he  was  concerned  with  the  analysis  of  synthetic  data 
from  three  contrived  models  and  the  performance  of  the  AID  algorithm, 
vis-a-vis,  the  binary  regression  approach  as  embodied  in  the  Multiple 


67 


Classification  Analysis  (MCA)  program.  The  data  set  consisted  of 
seven  variables  divided  into  49  categories,  on  each  of  2036  data  cases. 
The  models  used  to  generate  the  data  were: 

1)  An  additive  model  (trunk- branch) 

2)  A configural  model  in  which  advantages  substitute 
for  one-another  but  disadvantages  are  cumulative 
(upper  terminating  trunk-twig) 

3)  A configural  model  in  which  disadvantages  substitute 
for  one-another  but  advantages  are  cumulative 
(lower  terminating  trunk-twig) 

Runs  without  noise,  low  noise  (=10  percent),  and  high  noise  (=  20 
percent)  were  made  and  analyzed  to  see  if  they  would  result  in  the  same 
models  as  were  input. 

The  major  conclusions  of  the  investigation  as  presented  by 
Sonquist  (1970)  were: 

1)  An  additive  model  would  produce  a symmetric  tree 
(Sonquist  recognized  that  additivity  was  a sufficient 
condition  for  a symmetric  tree  but  did  not  address 
the  problem  of  necessity). 

2)  Small  but  universal  effects  may  not  be  discovered 
in  the  splitting  process. 

3)  Irrelevant  variables  which  are  correlated  with 
powerful  predictors  may  be  used  spuriously  in  the 
splitting  process. 

4)  The  presence  of  noise  (at  the  levels  considered)  did 
not  materially  affect  the  resulting  AID- Trees. 

^MCA  is  described  in  Appendix  A. 
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5}  One  evident-  of  interaction  is  a fairly  large  increase 
in  the  predictive  power  of  one  of  the  variables  after 
the  effects  of  some  of  the  other  predictors  had  been 
removed  by  the  splitting  process. 

6)  When  a nonsymmetric  structure  appears,  and  the  groups 
isolated  first  have  approximately  the  same,  low  or  high, 
means:  a clear  departure  from  additivity  has  occurred. 

7)  The  AID- Tree  was  effective  in  depicting  departures  from 
additivity  for  the  models  considered. 

8)  The  order  in  which  the  variables  are  used  in  the 
splitting  process  is  not  necessarily  reflective  of  their 
overall  importance  in  describing  the  process,  rather 
the  total  variation  explained  by  a variable  may  be  more 
indicative  of  .mporiance. 

9)  The  B statistics  calculated  and  output  by  the  tested  ver- 
sion cf  the  AID  pregram  were  virtually  meaningless  in 
depicting  the  relative  importance  of  the  individual 
variables. 

Iwo  important  aspects  of  this  investigation  deserve  special 
mention.  The  first  is  that  the  underlying  models  and  the  quality  o:  the 
data  set  were  known  on  an  a priori  basis.  The  second  is  that  a data 
set  of  approximately  40  points  per  predictor  category  was  available  and 
used  in  the  analysis. 

It  is  entirely  different  problem  to  sec  a known  model  in  the 
AID- Tree  than  it  is  to  infer  an  unknown  model  from  the  AID-Tree. 

Further,  the  stabilii  / of  ti  category  means  offered  by  a high  data/ 
predictor  ratio  helps  signi  icantly  in  discriminating  between  actual 


effects  that  are  due  to  the  predictors  and  the  spurious  effects  that  are 
due  to  the  idiosyncracies  of  the  data  sample. 

Sonquist  recognized  these  deficiencies  in  the  generality  of 
his  results  and  suggested  the  application  of  the  following  heuristic  pro- 
cedures when  analyzing  AID- Trees. 

1)  The  detailed  profiles  of  each  variable  used  in  a split 
(and  its  competitors  for  the  split)  should  be  examined 
■within  the  major  subgroups  throughout  the  tree. 

2)  All  variables  that  are  competitors  in  the  splitting 
process  should  be  examined  closely  for  dependence. 

If  two  variables  are  measures  of  the  same  aspect  of 
the  situation,  the  variable  that  is  least  supportable 
purely  on  logical  or  theoretical  grounds  should  be 
discarded. 

3)  The  residual  profiles  of  the  final  groups  should  be 
examined  to  see  if  effects  from  less  powerful  pre- 
dictors may  still  be  present  and  unaccounted  for  by  the 
splitting  process. 

In  regard  to  the  size  of  the  data  set  required  for  AID  analysis,  Sonquist 
(1970)  stated: 

Data  sets  with  a thousand  or  more  cases  are  necessary, 

otb  era'ise  the  power  of  the  search  process  must  be 

restricted  urastically  or  those  processes  will  carry 

one  into  a never-never  Land  of  idiosyncratic  results. 

(p.  15 

In  addition  to  generally  confirming  his  original  hypotheses 
relative  to  the  utility  of  AID  in  detecting  interaction  effects,  Sonquist1  s 
(1970)  investigation  did  provide  some  insight  into  the  problems  to  be 


encountered  in  running  AID.  For  these  problems  he  offered  the  following 
general  guidelines: 

1)  Do  not  categorize  the  variables  with  excessive  detail; 
five  to  seven  categories  per  predictor  should  be  the 
maximum  segmentation  of  the  predictors. 

2)  Categorize  the  variables  so  as  to  minimize  the 
problems  created  by  skewness  in  the  predictors:  if 

a predictor  is  skewed,  put  those  values  in  the  tail  of 
the  distribution  in  separate  categories. 

3)  Beware  of  skewness  in  the  criterion;  discard  outlier 
cases  or  perform  appropriate  transformation  of  scale 
on  them. 

4)  Perform  appropriate  initial  runs  with  the  split 

ter  mination  limits  disabled  in  order  to  detect  prob- 
lems such  as  measurement  or  coding  errors. 

Finally,  Sonquist  (1967,  1970)  proposed  a procedure  for 
using  AID  to  hypothesize  the  terms  of  the  model  and  then  using  MCA 
to  determine  the  proper  weights  of  the  individual  predictors.  The 
additional  analysis  with  MCA  was  espoused  since  the  AID  program  does 
not  produce  statistics  that  accurately  reflect  the  relative  weights  of 
the  individual  predictors. 

Problems  in  the  Application  of  AID  for  Hypothesizing  Policy  Models: 

The  application  of  this  "structure  searching"  technique  to 
the  hypothesis  of  policy  models  would  be  straightforward  except  for  two 
reser  vations--the  amount  of  data  required  to  obtain  stable  means  for 


the  subgroups,  and  the  lack  of  rules  for  inferring  an  "unknown"  model 
from  the  data  set. 

Given  the  objective  of  "data  conservatism",  as  presented 
in  Chapter  III,  and  the  difficulties  attendant  in  obtaining  large  quantities 
of  decision  data  in  operational  environments,  the  freedom  of  the  AID 
search  would  probabLy  have  to  be  somewhat  restricted.  Initial  efforts 
by  this  investigator  in  analyzing  the  loan  application  data  confirmed 
this  fact.  Splits  resulting  from  groups  of  less  than  10  cases  were 
generally  found  to  be  meaningless  and  uninterpr etable.  In  runs 
consisting  of  between  224  to  400  data  units,  with  approximately  50 
predictor  categories,  AID  displayed  a tendency  to  make  spurious  splits 
that  did  not  hold  up  under  cross-validation.  When  the  splitting  pro- 
cesses were  restricted  to  avoid  this  overfitting,  the  explained  vari- 
ation in  the  data  set  was  disappointingly  low  (<  50  percent). 

The  hypothesis  of  configural  judgment  models  based  solely 
on  the  splitting  process  was  found  to  be  less  than  successful,  as 
measured  by  the  lack  of  Improvement  in  the  predictive  capability  of 
the  resulting  configural  models.  The  transformation  from  various 
AID- Tree  structures  into  appropriate  terms  was  not  apparent.  "Con- 
tinuous" interaction  terms  of  the  form,  b.x.x.,  were  found  to  be  of 
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little  value  in  improving  the  fit  of  policy  models.  It  was  demonstrated 
that  the  number  of  categories  and  the  definition  of  these  categories 
could  materially  affect  the  appearance  of  the  AID- Tree.  Experimenta- 
tion with  various  configural  models  revealed  that  perfectly  symmetrical 
AID- Trees  could  be  generated  with  data  from  highly  configural  models. 
Hence,  additivity  was  found  to  be  a sufficient  condition  for  symmetry, 
but  not  a necessary  condition. 
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Although  the  AID-Tree  itself  proved  less  inter  pretable  than 
anticipated  in  inferring  configural  models  for  the  decision  process  of 
the  installment  loan  officers,  it  did  retain  the  advantage  of  displaying 
the  variables  and  their  relationships  in  a manner  that  was  readily 
comprehensible  and  believable  to  the  judges  involved.  In  all  cases  it 
identified  those  predictors  that  were  most  important  to  the  judges. 

The  predictors  used  in  the  early  splits  of  the  tree  appeared  to  be 
pivotal  elements  in  the  judges'  own  descriptions  of  their  thought 
pr oce  s s es . 

In  order  to  better  define  the  proper  configural  relationships, 
it  was  concluded  that  the  predictive  efficiency  and  the  correlation  of  the 
criterion  with  each  of  the  predictors,  within  each  subgroup  of  the  AID- 
Tree,  would  be  required  in  addition  to  the  AID-Tree  itself. 

This  same  conclusion  was  independently  reached  by  B.  M. 
Finifter,  of  Michigan  State  University,  who  published  the  description 
of  a proposed  addition  to  the  AID  program  in  September,  1971.  This 
addition  consists  of  a routine  to  collect  and  display  profiles  of  the 
Between  Sums  of  Squares/Total  Sum  of  Squares  (BSS/TSS)  from  the 
AID  splitting  process.  A tier malized  parameter  reflective  of  the 
predictive  ability  of  each  predictor  over  the  entire  AID-Tree  is  cal- 
culated and  designated  as  the  Potential  Explanatory  Power  (PEP) 
coefficient  (Finifter,  1 971,  1972). 

The  Development  of  the  AID4UT/ AIDTRE  Computer  Program: 

The  modification  and  extension  proposed  by  Finifter  was 
incorporated  into  an  improved  version  of  the  AID4  computer  program 
that  was  acquired  from  the  Air  Force  Homan  Resources  laboratory. 
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The  new  program  was  implemented  on  the  U.  T.  CDC-6600  computer 
system.  It  has  the  capacity  to  consider  unlimited  quantities  of  data 
with  up  to  80  predictors,  divided  into  up  to  700  predictor  categories. 
The  program  can  build,  cross- validate,  and  double  cross- validate 
AID-Trees.  and  it  can  produce  profiles  of  the  BSS/TSS  for  each  pre- 
dictor. A complete  description  of  the  modified  program  witn  Users* 
instructions  is  provided  in  Appendix  C. 


Application  of  AID4UT/ AIDTRE  in  Defining  Subspaces: 
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Although  the  addition  of  the  Finifter  modification  does 
provide  more  useful  statistics  for  discerning  unknown  structure  in 
the  data,  the  problem  of  overfitting  can  only  be  avoided  by  limiting 
the  number  of  splits  that  are  allow-ed.  In  general,  it  was  found  that  the 
limiting  subgroups  to  contain  no  less  than  5 percent  of  the  cases  would 
provide  splits  that  were  relatively  stable.  This  early  termination 
of  the  splitting  process  leaves  considerable  residual  variability  within 
the  final  subgroups.  Review  of  these  final  subgroups  revealed  that 
more  of  the  residual  variability  could  be  explained  by  various  pre- 
dictors, but  the  importance  of  any  particular  predictor  was  generally 
not  as  dominant  as  was  the  case  in  the  earlier  splits.  This  condition 
suggested  that  the  residual  variability  in  the  subgroups  might  better 
be  explained  by  a combination  of  the  predictors  instead  of  a binary 
split  on  a single  predictor.  In  repeated  applications  on  the  224  data 
units  for  the  decisions  of  the  installment  loan  officers,  the  criterion 
of  retaining  at  least  5 percent  of  the  total  in  each  of  the  final  subgroups 
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resulted  in  termination  of  the  splitting  process  within  10  to  15  splits. 

It  was  discovered  that  the  most  important  predictors,  in  terms  of  their 
configural  relationships  with  the  other  variables,  tended  to  be  involved 
in  the  early  splits.  Assessment  of  the  subgroups  resulting  from  these 
early  splits  indicated  that  they  were  often  homogeneous.  Within  the 
individual  subgroups,  different  combinations  of  the  predictors  appeared 
to  explain  the  variation. 

By  relating  this  phenomena  to  the  judges’  own  verbalizations 
of  their  policies,  and  reflecting  upon  the  findings  of  Hoffman  (1968), 
Bettman  (1969),  and  Sonquist  (1970),  the  concept  of  using  AID-Trees 
solely  for  the  purpose  of  defining  a series  of  subspaces,  and  then 
modeling  each  subspace  with  a local  model  evolved.  This  technique 
was  perceived  as  a means  of  taking  advantage  of  the  configural  effects 
of  the  predominant  variables  without  falling  into  the  trap  of  overfitting 
the  idiosyncracie s of  small  subsets  of  data. 

The  Criteria  for  Defining  Subspaces: 

The  decision  of  how  many,  and  which  subspaces  to  divide 
the  predictor  space  into  is  based  on  heuristic  rules  developed  during 
this  approach.  Experience  with  the  loan  officer  data  indicates  the  best 
criteria  appear  to  be  homogeneity  of  the  subgroup  and  the  predictive 
capability  (both  absolute  and  relative)  of  the  predictors  within  the 
subgroups. 

5 

Allowing  the  tree  to  split  into  smaller  groups  does  not  affect  the  first 
few  splits  since  the  tree  can  be  manually  truncated;  this  was  the  case 
with  the  loan  application  data. 
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The  logic  for  defining  appropriate  suhspaces  is  best  ex- 
plained by  an  example.  The  example  that  will  be  used  is  the  definition 
of  suhspaces  from  the  AID-Tree  for  one  of  the  five  installment  loan 
officers  whose  policies  were  captured  as  a part  of  this  research.  An 
annotated  AID-Tree  for  this  policy  model  is  shown  in  Figure  4-3; 
profiles  for  the  BSS/ TSS  values  for  all  16  variables  are  shown  in  Tables 
4-1  and  4-2.  The  data  shown  in  Table  4-1  reflects  the  maximum  between 
sum  of  squares  for  each  predictor,  in  a given  subgroup,  normalized  over 
the  Total  Sum  of  Squares  for  the  entire  data  set.  It  will  be  designated 
as  BSS/TSST.  Table  4-2  reflects  the  profile  in  which  the  maximum  be- 
tween sum  of  squares  for  each  predictor  has  been  normalized  over  the 
Total  Sum  of  Squares  within  the  subgroup  under  consideration  at  a given 
step  or  trial.  It  will  be  designated  as  ns.S/TSS(i).  These  two  profiles 
reflect  the  "overall"  and  "local"  predictive  capability  of  each  predictor. 
Comparison  of  these  profiles  are  very  helpful  in  revealing  configural 
and  non’ inear  effects  between  the  individual  predictors  even  when  one 
of  the  predictors  involved  is  not  actually  used  in  the  splitting  process. 

In  the  step-by-step  consideration  of  the  AID-Tree  that 
follows,  the  superscripts  refer  to  the  corresponding  entries  on  Tables 
4-1  and  4-2.  More  detailed  output  for  this  analysis  is  shown  in  Exhibit 
8 of  Appendix  B. 

Step  1.  Credit  Rating  has  the  largest  value  for  BSS/TSST, 

(cL  3 1 1 

Since  there  is  only  one  group  at  this  point,  entries  of  . 38  ’ appear 

in  both  Tables  4-1  and  4-2.  (The  next  largest  value  is  for  Financial 

b ^ 

Reference  which  has  . 17  ) Group  1 splits  into  groups  2 and  3.  Group  2 

is  a homogeneous  "twig"  with  a mean  criterion  value  of  .01.  The 

input  parameters  used  for  this  particular  run  of  AID4UT^  allow  no 

^The  setting  of  the  parameters  for  AID4UT  is  discussed  in  Appendix  C. 
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ANNOTATED  AID- TREE  FOR  EXAMPLE  PROBLEM 


FIGURE  4- 
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further  splitting  of  this  group  because  none  of  the  predictors  can 
explain  more  than  .01  of  the  total  variability. 

Step  2.  In  group  3,  Financial  Reference  has  the  largest 
BSS/TSST  of  . 17C  in  Table  4-1  and  a BSS/TSS(i)  of  . 28cl  in  Table  4-2. 
Note  that  the  BSS/ TSST  remained  the  same  for  this  variable  but  the 
BSS/TSS(i)  increased  markedly,  This  would  indicate  that  Financial 
Reference  is  immaterial  at  a Credit  Rating  level  of  "poor",  but  that  it 
is  the  single  most  important  variable  over  the  Credit  Rating  levels  of 
"unrated",  "medium",  or  "good".  Further,  note  that  the  predictive 
capability  of  Credit  Rating  decreased  by  a factor  of  two  on  the  "local" 

(BSS/TSS(i))  basis  after  the  split.  This  would  indicate,  that  there  is  a 
much  smaller  difference  between  the  upper  three  levels  of  Credit  Rating 
than  there  is  between  those  three  levels  and  the  lowest  level,  "poor". 

By  splitting  group  3 on  Financial  Reference,  a second  homogeneous 
"twig",  group  4,  is  formed  consisting  of  19  data  units  which  have  a 
mean  criterion  value  of  . 00.  Because  of  the  similarity  of  their  mean 
criterion  values,  group  2 and  group  4 are  combined  together  into 
subspace  1.  A one  term  model  (a  binary  indicator  of  group  member- 
ship) is  sufficient  for  this  subspace. 


Step  3.  In  group  5,  Bank.  Account  has  a DSS/TSST  of  .11 
.'(el) 


(e) 


and  a llSS/  TSS(i)  of  .29  ' . It  is  the  basis  for  splitting  group  5 into 

subgroups  6 and  7.  Neither  of  these  resultant  subgroups  is  as  homo- 
geneous an  were  the  previous  "twigs".  Kach  will  require  further 
modeling.  Considering  the  BSS/lSS(i)  profiles  for  group  6,  note  that 
both  Phone  ^ and  Credit  Rating^'’  jlmw  sizable  values.  However, 
the  group  b contains  only  30  data  units  and  a binary  split  on  any  one 
predictor  at  this  level  will  result  in  groups  that  contain  the  minimum 
allowable  number  of  data  units  and  are  still  nonhonmgeneous.  At  this 
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point,  it  is  better  to  model  group  6 with  a fir »i-order  model  containing 
both  Credit  Rating  and  Phone,  Thus,  group  6 becomes  subspace  2. 

Step  4.  In  group  7,  Credit  Rating  has  a BSS/TSS(i)  value 
of  . 10^^  which  is  only  slightly  larger  than  the  values  for  Restime^1^ 
and  Phone  However,  if  allowed  to  split  on  Credit  Rating,  a homo- 

geneous "twig"  will  result.  Group  9 has  53  data  units  and  a mean 
criterion  value  of  . 98.  It  comprises  subspace  3 and  is  modeled  by  a one 
term  model. 

c.  - T u T a ,(i.il)  ...  t(j.jl) 

Step  5.  In  group  8,  loan  Amount  , Reseat  , 

(k,  kl ) (m,  ml)  An.nl) 

Restime  , Marital  , and  L.oeal  , all  have  BSS/ Too 

values  reflecting  predictive  capability,  but  a binary  split  on  any  single 

one  of  these  variables  will  not  result  in  a homogeneous  group.  This 

group  becomes  subspace  4. 

This  walk-through  of  the  heuristic  logic  employed  in  defining 
subspaces  in  an  AID- Tree  brings  out  two  points  that  were  observed 
during  the  analysis  on  the  various  AID- Trees  in  this  i esearch.  First, 
reasonably  sized  groups  which  are  homogeneous  are  prime  candidates 
as  subspaces,  especially  if  they  are  split  off  in  the  very  early  stages 
of  the  AID- Tree.  Second,  in  those  parts  of  the  AID- Tree  where  the 
groups  are  not  homogeneous,  predictors  that  cannot  explain  in  excess 
of  10  percent  of  the  variability  within  the  subgroup  will  probably  not 
result  in  isolation  of  new  homogeneous  subgroups  of  substantial  size: 
in  this  case  it  appears  better  to  fit  local  "continuous"  linear  models. 

Building  Local  Models  Within  the  bub  spaces: 


Once  the  subspaces  are  identified,  the  data  in  the  BSS/ TSS(i) 
profile,  Table  4-2,  ran  be  of  further  use  in  hypothesizing  the  proper 
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terms  for  the  models.  In  formulating  the  models  for  the  individual 
subspaces,  there  are  several  possible  approaches.  If  there  is  sufficient 
data  available  (in  cases  where  at  least  one  data  unit  per  predictor 
category  is  available,  and  preferably  on  the  order  of  5-10  data  units 
per  predictor  category),  a binary  regression  model  could  be  built  by 
using  either  the  MCA  program  or  any  standard  regression  package.  In 
subspaces  where  the  number  of  data  units  is  not  amenable  to  using 
binary  or  categorical  models,  the  formulation  of  a "continuous"  first- 
order  regression  model  is  usually  a reasonable  alternative.  Under  the 
latter  formulation,  the  investigator  effectively  imposes  the  assumption 
that  a unit  change  in  the  predictor  value  will  produce  a fixed  (constant) 
change  in  the  criterion  variable.  He  makes  this  assumption  in  exchange 
for  using  fewer  degrees  of  freedom  and  predictors  in  the  model. 

In  the  definition  of  subspaces  and  local  models  for  the  five 
loan  officers,  various  alternative  formulations  were  attempted  at  the 
subspace  level.  For  example,  the  approach  of  combining  two  neigh- 
boring subspaces  into  a larger  subspace  and  formulating  a configural 
model  over  this  larger  subspace  was  compared  with  the  approach  of 
formulating  separate  first-order  models  over  the  original  subspaces. 

The  general  conclusion  drawn  from  these  efforts  was  that  the  approach 
of  using  first-order  models  over  the  individual  subspaces  was  superior. 

The  Determination  and  Evaluation  of  Model  Coefficients: 

The  task  of  obtaining  regression  coefficients  for  all  of  the 
models  investigated  was  performed  using  an  iterative  determination/ 
cross-validation  procedure.  A similar  procedure  was  originally 
suggested  by  a member  of  the  research  supervising  committee, 
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Dr.  E.  Jennings,  as  a method  of  evaluating  the  stability  of  alternative 

models.  The  technique  consists  of  the  random  selection  of  80  percent 

of  the  data  units  from  the  input  file,  generation  of  the  coefficients  using 

the  REGREJ  routine  of  the  EDSTAT-J  system  (Jennings,  1971),  and 

cross-validation  of  these  coefficients  with  the  remaining  20  percent  of 

the  data  units.  This  cycle  was  accomplished  10  times  and  resulted  in 

a profile  of  coefficients  for  each  predictor.  Whereas  Jennings  uses 

this  procedure  solely  to  assess  the  stability  of  various  models,  in  this 

research,  the  average  values  for  each  of  the  coefficients  were  actually 

used  in  the  model.  The  objectives  of  this  revised  procedure  were  the 

simultaneous  assessment  of  the  stability  of  the  model  and  the  reduction 

of  the  idiosyncratic  effect  that  any  outlier  data  units  would  have  on  the 

model  coefficient:.  The  average  coefficients  were  compared  with  the 

corresponding  coefficients  obtained  in  a single  fit  over  all  224  data 

units  for  each  of  the  five  judges.  They  were  found  to  be  very  similar. 

One  interesting  outcome  of  this  comparison  is  reflected  in  Table  4-3. 

2 

Note  that  the  average  values  for  predictive  efficiency  (R  ) for  the  10 

sets  of  coefficients,  resulting  from  the  iterative  scheme,  are  slightly 

2 

better  than  the  corresponding  values  of  R that  were  obtained  in  the 
single  fit  procedure.  This  would  suggest  that,  indeed,  some  of  the 
idiosyncratic  effects  of  outlier  points  may  have  been  overcome  by  the 
iterative  procedure. 
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TABLE  4-  3 

COMPARISON  OF  PREDICTIVE  EFFICIENCIES 
FOR  ALTERNATIVE  COEFFICIENT  DETERMINATION  PROCEDURES 
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Procedure 

J udge 

1 

2 

3 

4 

5 

Iterative 

Procedure 

. 670 

. 707 

. 680 

. 633 

. 632 

Single  Fit 
Procedure 

. 659 

. 695 

. 680 

. 614 

. 618 

Comparative  Results  of  Various  Modeling  Approaches: 
Installment  Loan  Officer  Models 


The  quality  of  the  various  models  investigated  in  this 

research  were  initially  measured  in  terms  of  the  standard  statistical 
2 

parameter,  R , and  the  cross- validity  from  one  subset  of  the  224  sample 
points  to  another  subset.  An  80/20  split  between  the  determination 
sample  and  the  cross-validation  sample  was  used.  The  average  cross- 
validity for  10  trials  per  model  of  the  first-order,  second- order , and 

local  models  are  shown  in  Table  4-4;  the  data  for  the  AID  splitting 

7 

process  consists  of  only  one  trial  per  AID- Tree.  These  data  reflect 
7 

The  process  of  using  an  80/20  split  in  the  AID  cross-validation  process 
was  found  to  influence  the  results  in  the  cases  of  limited  data  sets  since 
the  groups  in  the  subgroups  propagated  in  the  20  percent  sample  become 
very  small  and  susceptible  to  group  means  that  were  greatly  affected 
by  random  sampling  error.  The  cross- validity  coefficient  in  AID  is 
not  generally  monotonic  and  the  values  presented  in  Tabic  4-4  reflect 
the  maximum  cross- validity  attained  in  the  forced  splitting. 
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TABLE  4-4 

COMPARISON  OF  CROSS- VALIDITY  ON  SPLIT  SAMPLE 


Model 

1 

2 

J udge 
3 

4 

5 

First  Order 

. 579 

. 624 

. 634 

. 512 

. 538 

Second  Order 

. 649 

. 645 

. 653 

— 

— 

AID- Split 

. 440 

. 621 

. 675 

.670 

, 547 

Local  Model 

. 727 

. 659 

. 685 

.633 

. 558 

that  the  local  models,  derived  from  the  AID- Tree,  were  consistently 

better  than  the  first-order  or  the  second-order  models. 

In  actually  evaluating  the  goodness  of  the  various  modeling 

approaches  in  predicting  dichotomous  decisions  of  the  installment  loan 

2 

officers  hit  rate  is  a more  relevant  measure  of  goodness  than  R . Since 
each  of  the  models  resulted  in  a predicted  score  in  the  normal  range 

g 

from  . 00  to  1 . 0,  each  possible  value  in  this  range  could  be  used  as 
the  cut  score,  or  discrimination  point.  Above  this  point  the  value  of 
"l"(Yes)  would  be  assigned  to  the  data  unit,  and  below  this  point,  a 
value  of  "0"  (No)  would  be  given.  Generally,  at  each  possible  cut  point 
there  are  some  data  units  that  will  be  misclassified  (either  actual  0's 
given  the  value  of  1 , or  actual  l's  given  the  value  of  0).  The  hit  rate 
reflects  the  percentage  of  proper  classifications,  but  does  not  reflect 
the  type  of  misclassification  that  is  being  made.  Data  on  cross-valida- 
tion hit  rates  was  commensurate  with  that  shown  in  Table  4-4. 

g 

Actually  scores  slightly  smaller  than  . 00  and  slightly  larger  than  1. 00 
can  be  obtained  but  in  those  cases  the  scores  are  rounded  to  . 00  or 
1.00  respectively. 
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The  indifference  of  the  magnitude  of  hit  rate  to  type  of  error 
is  extremely  important  since  the  decision  maker  may  be  more  concerned 
with  one  type  of  error  than  he  is  with  the  other  type.  Hence,  the  per- 
formance of  the  model  over  a range  of  possible  cut  points  is  important. 
Figure  4-4  schematically  depicts  the  profiles  of  hit  rates  for  first-order 

COMPARATIVE  PROFILES 
OF  HIT  RATES 


CUT  SCORE 

FIGURE  4-4 


and  local  models  for  the  installment  loan  officer  data.  It  shows  that 
even  if  the  maximum  hit  rate  of  the  first-order  model  is  marginally 
better,  the  profile  is  generally  inferior  and  the  marginal  superiority 
prevails  over  only  a very  small  range  of  cut  scores.  Thus,  the  single 
parameter  embodied  in  either  the  hit  rate  or  the  cross-validation 
coefficient  is  probably  not  the  best  criterion  for  determining  overall 
su;  eriority,  but  rather,  its  magnitude  and  profile  should  be  considered 
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jointly.  Figures  6-14  through  6-18  in  Chapter  VI  reflect  that  both  the 
maximum  hit  rates  and  the  profiles  were  superior  for  the  local  models 
in  the  80/20  cross-validation  process. 

A completely  separate  cross-validation  effort  using  150 
data  units  collected  during  a different  time  period  was  also  performed. 
The  maximum  hit  rates  for  the  first-order  models  were  slightly  superior 
to  those  for  the  local  models  in  this  test.  However,  this  reversal  can 
be  traced  to  a significant  shift  in  the  characteristics  of  the  data  set  and 
a shift  in  the  policy  of  the  judges.  The  full  details  of  this  test  are  dis- 
cussed in  Chapter  VI. 

The  Valenzi  Data,  an  Independent  Example 

Whereas  the  approach  of  defining  subspace.s  and  local  models 

was  conceived  as  a result  of  the  initial  analysis  of  the  policies  of  the 

loan  officers  and  was  developed  primarily  using  this  data,  it  might  be 

criticized  as  being  useful  only  in  situations  with  binary  criterion 

values.  An  insight  into  the  generality  of  the  local  modeling  approach 

9 

was  sought  by  applying  it  to  an  independent  set  ol  data  for  which  a 
continuous  criterion  existed.  The  data  used  for  this  analysis  was 
published  by  Valenzi  (1970).  The  data  consisted  of  the  judgments  made 
by  employment  counselors  relative  to  the  chances  of  applicants  being 
hired  for  secretarial  jobs.  Five  tricotomous  predictors  (typing  speed, 
shorthand  speed,  experience,  education,  and  social  skills)  were  used 
to  describe  each  applicant.  The  criterion  was  a scalar  value  from  0 to 

9 

Independent  in  this  context  denotes  that  the  investigator  had  not  pre- 
viously analyzed  the  data  and  was  not  able  to  intuit  a good  model  r.s  a 
result  of  familiarity  with  the  process  or  the  data. 
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10.  A full  3 factorial  experimental  design  with  two  replications  per  cell 
was  analyzed  with  both  ANOVA  and  multiple  regression  models.  Valenzi 
attempted  to  demonstrate  and  evaluate  configural  cue  use  by  four  judges; 
the  analysis  and  comparisons  in  the  following  discussion  considers  only 
one  judge,  Judge  4. 

Vaienzi  published  the  following  results  for  Judge  4: 

1 . ANOVA  results  with  main  effects  and  all  two-way 

interactions  resulted  in  7 3 percent  of  the  variability 
being  explained  by  the  main  effects  and  14.  5 percent 
being  explained  by  the  second-order  configural  effects. 
A total  of  87.  5 percent  of  the  variability  was  explained 
with  some  41 . 5 percent  beinr  explained  by  the  educa- 
tional main  effect. 

2.  The  second-order  multiple  regression  model  resulted 
in  60  percent  of  the  variation  being  explained  by  the 
first-order  terms  and  9.7  percent  being  explained  by 
the  second-order  terms. 

3.  The  largest  interaction  identified  by  either  of  these 
processes  was  Education  by  Social  Skills  which 
explained  8.  5 percent  of  the  variability  in  Valenzi's 
ANOVA  model  and  5,  4 percent  in  his  second- order 
regression  model. 

4.  The  importance  ranking  for  the  predictors  given  by 
Judge  4 did  not  match  the  rankings  derived  from  the 
statistical  analysis  models. 

Vaienzi  concluded  that  Judge  4 did  exhibit  configural  cue 
use  and  that  ANOVA  was  a better  approach  to  detecting  it  than  were 


standard  multiple  regression  techniques  or  the  "a  priori  partialling 
out"  of  the  configural  variance  suggested  by  the  judge's  verbalization 
of  his  policy. 

The  results  obtained  by  applying  the  AID4UT/AIDTRE  com- 
puter programs  and  the  local  modeling  approach  to  the  same  data  were: 

1.  An  AID-Tree  with  14  final  groups  explained  89  percent 
of  the  variability.  Using  an  80/20  split,  a cross- 
validity of  85  percent  was  achieved. 

2.  The  first  split  in  the  AID-Tree  was  made  between  the 
lowest  and  the  upper  two  levels  of  Education.  This 
split  explained  40.  1 percent  of  the  variability  and 
resulted  in  a homogeneous  "iwig"  which  included 

162  data  units  {1/3  of  the  total  data  set).  This  group 
had  an  average  criterion  value  of  . 01  on  a scale  from 
0 to  10. 

3.  The  most  important  variable  for  the  other  324  cases 
was  social  skills.  Its  interaction  with  all  other 
variables  was  readily  detectable  in  the  detailed  output 
from  AID4UT. 

4.  Definition  of  two  sub  spaces  on  the  basis  of  the  first 
split  and  the  use  of  the  following  local  models, 
(bubspace  1 included  all  cases  with  the  lowest  level 
of  Education  and  subspace  2 included  all  others.  ) 

a,  Y - b t Y H e:  for  subspace  l 

b 

b.  Y - Uj  (social  skill ) (typing) 

+ b^  (social  skill)(shorthand) 

H b^  (social  skill)(education) 
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+ b (social  sk.ill)(exper  ience)  +et 
4 

for  subspace  2 

resulted  in  a 6 predictor  model  which  explained  81. 8 
perc<  nt  of  the  variance,  a gain  of  12.  1 percent  relative 
to  the  1 5 predictor  second-order  regression  model  used 
by  Valenzi. 

5.  Changing  the  model  over  subspace  2 to  a binary  regres- 
sion model  which  considered  all  two-way  interactions, 

90  percent  of  the  variation  was  explained. 

6.  The  rank  order  of  the  variables  as  determined  by  the 
profile  of  BSS/TSS(i)  from  the  AID4UT  program  was 
more  consistent  with  the  judge's  verbalizations  than 
the  corresponding  ranking  achieved  by  Valenzi  with 
ANGVA  and  regression  analysis.  Table  4-5  reflects 
the  comparison.  A complete  reversal  of  the  ranks  of 
the  first  four  variables  arose  from  ANOVA.  Only 
social  skills  was  out  of  sequence  with  AID4UT. 


TABLE  4-5 

COMPARISON  OE  RANKED  IMPORTANCE  OF  PREDICTORS 


Rank  Judge's  Verbalization  ANOVA  AID4UT 


1 

shorthand 

education 

social  skills 

2 

typing 

social  skills 

shorthand 

3 

social  skills 

typing 

typing 

4 

education 

shorthand 

education 

5 

exper ience 

experience 

experience 

The  implications  of  the  comparison  oi  the  results  achieved 
on  the  Valenzi  data  are  clear.  The  use  of  the  AID  algorithm  to  isolate 
homogeneous  subgroups  over  which  better  fitting  local  models  can  be 
applied  is  potentially  superior  in  situations  other  than  those  with 
dichotomous  criteria.  The  superiority  of  the  technique  is  not  solely  a 
function  of  having  "lived"  with  the  data  and  the  decision  process  for  a 
prolonged  period  of  time.  In  this  independent  example,  the  model 
achieved  was  simpler  and  easier  to  reconcile  with  the  verbalized  policy 
of  the  j udge . 


Summary  of  Guidelines  for  Applying  AID4UT/AIDTRE  in  Policy  Capturing: 


The  experience  gained  thus  far  in  applying  A1D4  UT/ AIDTR  E 
to  Policy  Capturing  is  extensive  in  the  sense  that  numerous  models  have 
been  attempted  and  produced,  but  limited  in  the  sense  that  these  models 
were  all  for  a particular  decision  process,  except  for  the  brief  demon- 
stration on  the  Valenzi  data. 

Table  4-6  presents  a set  of  guidelines  that  should  make 
future  applications  of  the  AID  algorithm  and  the  local  modeling  technique 
more  efficient  and  effective.  These  guidelines  reflect  the  experience 
of  Sonquist  (1910)  as  well  as  the  experience  gained  in  the  current  research 
effort.  They  are  directed  at  using  AID4 UT/ AIDTRE  for  the  definition  of 
subspaces  and  local  models  when  there  is  limited  data  available. 

These  empirical  guidelines  for  applying  AID  are  somewhere 
between  the  stages  of  artistic  and  heuristic.  This  investigator  doubts 
whether  such  guidelines  will  become  more  than  heuristic  in  the  fore- 
seeable future.  Considering  that  AID  is  a model  seeking  technique  and 
its  functions  is  to  uncover  relationships  in  data  sets  with  unknown 
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structure,  it  is  improbable  that  precise,  deterministic  criteria  for 
defining  subspaces  or  for  specifying  exactly  which  predictors  are 
important,  can  be  defined.  If  enough  information  were  Known  about 
the  data  set  to  do  this,  the  very  need  for  the  model  seeking  technique 
would  be  doubtful. 


TABUS  4-6 

GUIDELINES  FOR  EMPLOYING  AID4UT/AIDTRE 
IN  BUILDING  POLICY  MODELS 

PREPARATION  OF  DATA: 

1.  Segment  predictors  into  3-8  categories.  A lower  limit  of  at  least 
5 data  units  per  category  should  be  sought. 

2.  If  the  criterion  variable  is  badly  skewed  either  divide  the  data  set 
into  subgroups  or  make  a transformation  on  the  criterion  such  as 
Y1  = log  Y. 

3.  Obtain  a contingency  table  for  the  predictors.  An  association 
measure  such  as  the  Goodman  and  Kruskal  \ as  produced  by 
Anderberg's  (1971)  GCORR  program  is  helpful.  Note  those 
variables  with  very  high  associations  since  they  may  be  measures 
of  the  same  phenomena  and  substitute  for  each  otherin  the  split- 
ting process.  In  such  cases  one  of  the  variables  should -be  eli- 
minated from  consideration  as  a predictor. 

INITIAL  RUNS  OF  AID4UT/AIDTRE 

1.  Run  AID4UT  with  all  stopping  parameters  disabled.  (See  Appendix 
C)  . This  produces  a "configural"  model  which  provides  an  upper 
limit  on  the  capability  of  the  set  of  predictors  under  consideration 
to  predict  the  variability  in  the  data.  For  large  data  sets  with 
many  predictor  categories,  this  run  may  be  very  time  consuming 
and  setting  the  minimum  subgroup  size  to  . 01  of  the  total  set  may 
be  advisable. 

2.  Ref'ine/collapse  the  predictor  categories  if  possible.  If  two  or 
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more  categories  of  a predictor  never  end  up  in  different  subgroups 
as  a result  of  a split  they  can  be  combined  into  one  category. 

3.  Analyze  the  residuals  fur  obvious  coding  errors.  Such  errors  are 
usually  subtle,  but  can  affect  later  results  considerably. 

ANALYSIS  RUNS  OF  AID 4 UT/ AID TR E 

1.  After  corrections  and  modifications  from  the  initial  runs  are 
completed,  rerun  the  analysis  with  increased  stopping  parameters. 
In  data  limited  cases  where  less  than  19-20  data  units  per  pre- 
dictor category  exist,  set  split  r .'duceability  . 01  and  the  minimum 
group  size  to  > . 05  of  the  original  sample  size. 

2.  Analyze  tree  to  identify  large,  homogeneous  "twig"  groups  indi- 
cated by: 

a.  Mean  criterion  values  of  > . 9 or  < . 1 for  dichotomous 
criteria 

b.  Standard  deviations  of  the  criterion  values  of  < .2  of  the 
standard  deviation  of  the  o-iginal  group  for  continuous 
c*.  itcria. 

3.  Study  the  moan  criterion  values  for  each  predictor  category  in  each 
subgroup  for  the  first  4 or  5 splits.  Pertinent  point.-;  art:: 

a.  The  difference  in  the  magnitude  of  the  mean  criterion 
values  in  two  subgroups  resulting  from  a split  reflect 
the  "main  effect"  of  the  split  variable  over  the  parent 
group.  Of  particular  importance  are  the  trends  of  the  mean 
criterion  values  thr  mghout  the  AID- Tree.  Plots  of  the 
mean  criterion  values  versus  the  predictor  categories  for 
each  of  the  subgroups  in  the  AID- Tree  will  portray  these 
trends  If  the  trends  are  similar  then  the  predictor  is 
likely  to  be  independent  of  the  variables  used  in  tin.  previous 
splits;  if  the  trends  for  a predictor  vary  greatly,  there  may 
be  interaction  present  between  the  predictor  and  the  variables 
used  in  the  previous  splits. 

b.  If  the  standard  deviation  of  the  criterion  values  for  a given 
predictor  category  is  near  zero  and  the  category  is  small 
(<  , 05  of  total  sample)  this  represents  a homogeneous 
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group  which  may  not  bo  automatically  identified  by  the  AID- 
Treo  because  of  its  size  and  the  intervention  of  more  power- 
ful predictors  in  the  early  stages  of  the  splitting  process.  If 
there  is  a plausible  theoretical  reason  for  this  group  it  could 
be  removed  from  the  data  set  in  future  analysis. 

Analyze  the  DSS/TSS  profiles  to  ascertain  predictive  capability 

of  the  individual  predictors  over  the  AID- Tree.  Pertinent  points 

are: 

a.  If  lib'b/Tb'S(i)  is  greatly  (a  factor  of  2)  different  in  the  groups 
resulting  from  a split,  two  situations  may  exist;  either  there 
is  an  interaction  between  this  variable  and  the  split  variable 
or  they  are  measures  of  the  same  phenomena.  A check  of 
the  contingency  table  would  indicate  if  they  are  highly 
correlated  and  it  substitution  in  likely.  If  the  Kb'S/  TSS(i) 
for  the  split  variable  changes,  either  groups  with  a single 
category  of  that  predictor  have  been  created  or  there  is  a 
higher  order  effect  for  the  predictor. 

b.  If  the  Ubb/Tbb(i)  value  for  a predictor  remains  the  same 
after  it  lias  been  involved  in  a split,  but  it  loses  the  next 
split,  a "continuous"  interaction  (as  opposed  to  the  discrete 
interactions  represented  by  discrete  hyperpluues)  may  be 
present.  Separate  analyses  for  each  of  the  categories  of 
the  predictor  involved  would  result  m similar  All)- Trees 
for  each  category,  however,  the  mean  criterion  response 
values  for  cor  re  j ponding  subgroups  in  the  various  AID- 
Trees  would  be  different. 

c.  If  the  KbS/Tbb(i)  for  a predictor  remains  small  . Oh),  but 
relative  constant  over  the  entire  AID- Tree,  the  predictor 
may  have  a small  universal  ellect  and  could  Lie  included  in 
each  local  model. 

d.  Comparison  of  the  1SSS/ Tbb'(i)  and  DSb/'i’SbT  profiles  will 
give  further  indications  if  a variable  lias  a universal  or 
only  a local  effect.  Those  variables  which  appear  very 
low  in  importance  on  both  scales  can  probably  be  discarded 
from  the  predictor  set. 

e.  The  identification  of  important  variables  for  the  various 
subgroups  can  be  accomplished  from  the  profile  of  Kb'S/ 
TSS(i).  Listing  the  most  important  variables  on  the  skeleton 
version  of  the  All)-  Tree  will  provide  a visual  impression  of 
groups  in  which  the  same  predictors  a e effective. 


DEFINITION  OF  SUI3SPACE  AND  LOCAL  MODELS 


1 , Subspaces  can  be  identified  on  three  bases. 

a.  Homogeneity  relative  to  the  criterion  value. 

b.  Consistency  relative  to  the  predictors  that  are  important 
within  the  subgroup. 

c.  Consistency  relative  to  the  effects  that  the  individual  pre- 
dictors have  in  the  local  subgroups. 

2.  If  there  are  no  homogeneous  groups  identifiable  on  the  basis  of  the 
above  character  is- ties,  the  total  predictor  spice  should  be  repre- 
sented with  one  "continuous"  model.  In  this  case  the  hypothesis  of 
"continuous"  interaction  terms  of  the  type,  b.  x x , can  bo  accom- 

U 1 J 

plished  by  considering  the  JiSS/TSST  profile. 

3.  If  there  are  homogeneous  subspaces  within  the  AID- Tree,  the 

information  from  the  splitting  process  provides  the  basis  for  the 
generation  of  these  .subspaces  directly.  The  residuals  list  may 
be  used  to  identify  a ml  physically  separate  the  data  cases  in  the 
various  subspaces  for  separate  analyses.  Alternatively,  the 
logical  capabilities  of  the  computer  can  be  used  to  define  binary 
sub  space  multipliers.  CAUTION:  It  is  extremely  easy  to  define 

overlapping,  sub^paces  it  the  split  process  of  the  AID- Tree  is 
not  followed  precisely. 

4.  Once  the  subspaces  art-  identified,  the  predictors  within  each 
local  model  are  hypothesized  on  the  basis  of  the  HSS/  TSS(i) 
profilm  If  sufficient  data  units  exist,  binary  models  can  be 
formulated.  Otherwise,  "continuous"  lirst-ordcr  regression 
model:;  appear  to  be  the  best  approach.  Those  variables  not 
inherently  monotonic  should  be  checked  for  this  property  over 

the  subspace  undi  r consideration.  This  is  an  implicit  assumption 
of  "continuous"  formulation  and  the  L5SS/TSS  values  do  not  nec- 
essarily reflect  this  restriction. 

GENERATION  OF  REGRESSION  COEFFICIENTS 

Any  available  regression  package  can  be  used  to  generate  the 
appropriate  regression  coefficients.  The  data  transformation  capabi- 
lities available  in  the  DA1RAN  feature  of  the  EDSTAT-J  package  are 
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particularly  compatible  with  the  logical  definition  of  subspaces  on 
the  computer.  The  use  of  the  iterative  coefficient  determination/ 
cross-validation  approach  i:.  recommended  since  it  allows 
assessment  of  the  coefficient  stability  in  the  process  of  model 
building. 
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CHAPTER  V:  BACKGROUND,  ENVIRONMENT,  AND 

CHRONOLOGY  OF  CASE  STUDY 


Introduction; 


y 


This  chapter  provides  the  scenario  and  a brief  history  of  the 
activities  of  this  research  that  were  specifically  related  to  capturing  and 
evaluating  the  policies  of  a group  of  installment  loan  officers.  The 
results  and  implications  are  then  presented  in  Chapter  VI. 

A Brief  Resume  of  Numerical  Credit  Scoring: 


The  application  of  numerical  rating  systems  to  credit 
evaluation  was  first  proposed  by  Durand  (1941)  in  a study  of  several 
hundred  accounts.  He  used  discriminant  analysis  to  categorize  those 
accounts  on  the  books  as  "good"  and  "bad"  on  an  "ex-po,st"  quality* 
basis.  His  study,  and  others  that  followed,  generally  indicated  the 
potential  utility  of  numerical  credit  scoring  systems  in  screening  out 
bad  accounts.  However,  there  was  no  meaningful  implementation  of 
any  such  systems  until  about  I960.  Myers  and  Forgy  (1963)  attribute 
this  lack  of  implementation  to: 

1)  a natural  reluctance  on  the  part  of  the  experienced 
credit  executive  to  abandon  the  time  honored  "judg- 
mental" approach  in  favor  of  the  newer  and  relatively 
untested  quantitati vc  methods; 

2)  the  inability  of  statisticians  to  develop  "foolproof" 

*"ex-post"  quality  is  based  on  collection  experience  and  "ex-ante" 
quality  is  based  on  prospective  risk.  The  decision  process  considered 
in  this  research  is  based  cn  the'Vx -ante"  quality  of  the  loan  application. 
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rating  systems  which  consistently  identify  poor  credit 
risks  accurately  enough  to  result  in  substantial  net 
savings  by  the  credit  operation; 

3)  the  difficulties  involved  in  utilizing  any  such  effective 
rating  system  in  the  operating  situation;  and 

4)  unwillingness  on  the  part  of  the  statistical  consultants 
to  invade  the  domain  of  the  credit  manager  and  do  the 
selling  job  necessary  to  transform  such  an  idea  into  a 
successful  and  useful  operating  tool.  (p.  120) 

In  their  own  study,  Myers  and  Forgy  (1  963)  set  about 
analyzing  various  statistical  modeling  techniques  for  scoring  loans  on 
mobile  homes.  Again  they  used  the  common  procedure  of  scoring 
"ex-post"  quality  and  did  not  consider  the  judgment  process  of  the 
credit  executive  nor  did  they  demonstrate  the  operational  feasibility 
of  the  system.  They  did  make  one  advancement  by  validating  their 
models  by  means  of  cross-validation. 

The  expanding  availability  and  use  of  computers  within  the 
credit  industry  during  the  tnid-1960's  propagated  renewed  interest  in 
numerical  scoring  systems  and  considerable  debate  as  to  their  utility 
has  transpired  in  financial  journals  such  as  Credit  World  and  Ba nker ' s 
Monthly . Continued  research  in  the  area  has  proceeded  under  the  spon- 
sorship of  the  National  Bureau  of  Economic  Research,  the  Credit 
Research  Foundation,  the  First  Pennsylvania  Bank  and  Trust,  and 
Morris  Plan  Company  of  California. 

By  1968  some  35%  of  the  top  200  banks  in  the  United  States 
were  using  numerical  credit  scoring  of  some  form,  and  another  33%  were 
considering  such  systems.  Of  those  banks  having  systems,  some  80% 


had  been  borrowed  by  one  institution  from  another.  The  borrowing  of 
scoring  systems  was  attributed  to  the  extensive  analytical  effort 
involved  in  setting  up  a "tailored"  system,  and  was  responsible  for 
significant  implementation  problems.  (Wilt  and  Tierney,  1968) 

\s  a result  of  these  problems  brought  about  by  using  borrowed  systems, 
the  credibility  and  ut’lity  of  numerical  credit  scoring  has  suffered  even 
more  in  the  eyes  of  the  critics. 

In  a review  of  the  first  decade  of  use  of  numerical  scroing 
systems,  Zaegel  (1971)  noted  that  the  benefits  from  use  of  such  systems 
could  not  be  accurately  measured,  but  that  losses  were  down  approxi- 
mately 20%  in  situations  where  the  systems  were  being  used.  He  did 
note  that  many  improvements  could  be  made  especially  with  regard  to 
the  time  it  took  to  establish  and  validate  a system,  which  requires  about 
five  years  for  the  loan  turnover  cycle  when  the  system  is  based  on 
"ex-post"  quality. 

Although  brief,  this  resume  sets  the  stage  for  the  research 
into  applying  Policy  Capturing  techniques  in  defining  credit  scoring 
models  that  are  more  "personal",  "flexible",  "understandable",  and 
hence,  more  likely  to  be  implemented  in  operational  environments. 

The  Decision  Process  Considered  in  this  Study: 

The  primary  decision  process  modeled  in  this  research  is 
that  of  the  installment  credit  officer  in  making  judgments  of  whether 
to  grant  or  to  deny  a loan  to  an  applicant  based  on  information  contained 
in  a written  application.  The  decisions  being  modeled  are  limited  to 
those  involving  "dealer  paper",  i.  e.  , loan  applications  forwarded  to 
the  lending  institution  as  a result  of  a dealer  contract.  Under  a dealer 


contract,  the  lending  institution  provides  the  financing  for  a merchant's 
sale  on  an  individual  account  basis,  i,  e.  , they  agree  to  buy  the  install- 
ment contracts  from  a merchant  on  an  individual  basis  if  the  applicant 
and  the  account  meet  the  lending  institution's  standards  of  acceptability. 
Unlike  more  general  credit  decisions  in  which  the  interest  rates  and 
other  particulars  of  the  loan  are  negotiable  parts  of  the  decision,  the 
dealer  contracts  under  consideration  carry  a fixed  interest  rate  and 
provide  that  the  merchandise  will  serve  as  collateral.  Further,  the 
amount  and  duration  of  the  loan  are  not  actually  considered  as  items  of 
negotiation,  but  as  parameters  of  the  loan.  Thus,  the  decision  is 
essentially  uni-dimensional  in  that  only  a binary  decision  of  "accept" 
or  "reject"  is  made. 

A flow  diagram  of  the  "dealer  paper"  process  is  shown  in 
Figure  5-1.  Note  that  no  personal  contact  exists  between  the  loan 
applicant  and  the  credit  officer  and  the  decision  is  made  solely  upon  the 
basis  of  the  input  data  contained  on  the  written  application  and  the 
applicant's  credit  rating  that  is  obtained  from  the  local  credit  bureau. 

The  original  application  form  (Exhibit  1,  Appendix  13)  is 
filled  out  by  the  clerical  staff  at  the  dealer  location  and  is  then  tele- 
phoned to  the  lending  institution.  There  the  clerical  staff  of  the 
installment  loan  department  transcribes  the  information  and  calls  the 
local  credit  bureau  for  a credit  report.  Upon  receipt  of  this  report, 
the  information  is  attached  to  the  application  and  forwarded  to  an 

available  loan  officer  for  his  judgment,  If  he  is  a senior  loan  officer 
__  - 

In  special  cases  an  "accept  with  full  recourse"  is  made  in  which  the 
lende>-  grants  a loan  he  would  ordinarily  reject  if  the  merchant 
guarantees  payment  in  case  of  default.  In  this  study,  these  are  consi- 
dered rejects. 
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his  judgment  is  generally  conclusive,  but  if  he  is  junior,  he  normally 
makes  a recommendation  and  obtains  concurrence  from  one  ol  the  senior 
loan  officers.  When  the  decision  ’ made,  the  dealer  is  notified.  If 
the  loan  is  accepted,  he  executes  the  contract  and  releases  the  merchan- 
dise to  the  customer;  at  this  point  the  responsibility  fur  maintenance  of 
the  account  and  collection  of  the  loan  devolves  to  the  lender.  If  the 
decision  is  denial,  the  dealer  must  either  seek  financing  elsewhere, 
carry  the  note  himself,  or  vitiate  the  sale. 

The  entire  process,  from  sale  to  a decision  on  the  loan, 
involves  approximately  four  clerks,  two  transcriptions  ol  the  data,  and 
the  judgment  of  one  or  more  loan  officers.  II  requires  from  one  hour 
to  several  days.  For  purposes  of  later  discussion,  the  "system"  will 
include  those  elements  from  sale  until  notification  of  decision  and  the 
"credit  system"  will  include  those  elements  from  receipt  ol  the 
application  by  the  lending  institution  until  the  ratification  of  the  decision. 

The  Lending  Institution: 

The  installment  credit  department  participating  in  this  study 
is  part  of  i.mur  Savings  and  Loan  Association  located  in  Austin,  Texas. 
The  department  us  Undergoing  a transition  period  of  rapid  growth.  In 
the  last  three  years  their  outstanding  installment  accounts  have  in- 
creased over  500%  from  less  than  million  to  over  $10  million.  In 
1 970  and  19V1  tiiey  began  to  concentrate  their  efforts  on  dealer  contracts 
and  that  source  of  business  amounts  to  approximately  68%  of  their 
current  volume.  They  are  currently  processing  approximately  600 
applications  per  month  with  a stall  of  five  loan  officers  and  a clerical 
staff  of  1 5 people. 
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The  Decision- Maker s : 

The  decision- maker s who  acted  as  the  principle  judges  in 
this  research  were  the  five  installment  loan  officers  of  Lamar  Savings. 
Two  other  experienced  loan  officers  from  other  lending  institutions  in 
Austin  also  participated  in  the  effort  to  assess  the  transferability  of 
the  policy  models.  A short  background  sketch  of  each  judge  is  given 
below : 

1)  Judge  fll  is  an  Assistant  Vice-President  and  is  the 
number  two  man  in  the  installment  loan  department. 

He  has  nine  years  experience  in  installment  lending 
and  has  been  with  this  department  for  3.  5 years. 

2)  Judge  H2  is  a Vice-President  and  is  the  manager  of 

i 

the  installment  loan  department.  He  has  14  years 
experience  and  has  been  with  the  department  for  three 
years.  He  has  had  previous  experience  with  numeri- 
cal credit  scoring  systems. 

3}  Judge  #3  is  an  Assistant  Vice-President  and  is  third 
senior  man  in  the  department.  He  has  seven  years 
experience,  most  of  which  has  been  under  the  super- 
vision of  Judge  HZ  and  concentrated  in  the  area  of 
collection  of  delinquent  accounts.  Hu  has  been  in  the 
department  two  years. 

4)  Judge  II 4 is  a loan  officer  and  has  three  years  experi- 
ence, all  in  this  department. 

Judge  Hb  is  a loan  officer  and  has  two  years  experience, 
all  with  this  department. 
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6)  Judge  #6  was  recently  the  manager  of  a competitive 
lending  institution  in  Austin  and  was  hired  into  the 
department  near  the  end  of  the  project.  He  has  had 
experience  with  dealer  contracts  and  numerical  credit 
scoring  and  a total  of  four  years  experience. 

7)  Judge  #7  is  the  manager  of  a local  credit  union  and 
has  14  years  experience.  His  organization  does  not 
have  any  dealer  contracts  and  they  do  not  currently 
use  numerical  scoring  techniques. 

1'wo  Functional  Objectives  of  the  Policy  Models,  Tentative  Approvals 
and  Surrogate  Judgments: 

In  reference  to  the  decision  process  being  considered, 
the  objectives  of  the  modeling  effort  arc  to  provide  decision  models 
that  can  be  used  both  for  screening  the  loan  applications  on  a "tentative 
approval"  basis  and  also  pre-scoring  the  applications  on  a "surrogate 
judgment"  basis.  The  difference  between  these  two  functions  lies  in 
the  a vailability /non-a vniJability  of  the  credit  rating  for  use  as  a pre- 
dictor . 

As  depicted  in  Figure  5-1,  the  a va' 'ability  of  a credit 
rating  generally  requires  an  input  from  the  local  credit  bureau.  At 
present,  this  input  is  obtained  by  telephone  and  is  only  available 
during  normal  working  hours  from  Monday  through  Friday.  If  a 
merchant  operates  on  the  basis  of  six  or  seven  days  a week,  this 
represents  a significant  impediment  in  his  ability  to  sell  and  obtain 
timely  financing  for  his  merchandise.  No  action  can  be  taken  on  a 
loan  application  taken  after  4 p.  m.  Friday  until  at  least  8 a.  m.  the 
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following  Monday  and  the  merchant  must  either  delay  delivery  of  the 
merchandise  or  assume  the  risk  of  possibly  having  to  retrieve  it  if  the 
loan  is  denied.  This  delay  represents  a tangible  economic  consideration 
to  the  dealer  which  he  would  like  to  minimize,  especially  in  the  case  of 
those  customers  who  appear  to  qualify  for  a loan  on  the  basis  of  the 
other  information  available  on  the  application  form. 

Whereas  the  credit  rating  is  the  most  important  single 
variable  considered  in  the  loan  officers1  decisions,  they  are  generally 
unwilling  to  make  a final  commitment  without  it.  However,  it  has  been 
found  that  people  with  r.ood  credit  ratings  also  generally  score  well  on 
the  other  predictors.  The  problem  is  to  determine  if  a policy,  based 
on  only  the  other  predictors,  can  be  defined  that  will  identify  a suf- 
ficient number  of  good  accounts  to  justify  its  use  without  involving 
undue  risk.  If  such  a policy  can  be  found,  the  credit  executives  would 
be  willing  to  assume  some  increased  risk  in  order  to  provide  better 
service  to  the  merchant.  In  this  regard,  a "tentative  approval"  would 
be  an  approval  given  in  the  absence  of  credit  rating  information  which 
might  require  some  modification  of  the  contract  such  as  "full  recourse" 
if  the  credit  rating  was  later  determined  to  be  unacceptable. 

The  pre- scoring  or  "surrogate  judgment"  application  of  the 
policy  models  would  be  accomplished  with  the  use  of  the  credit  rating. 

In  this  case,  the  policy  model  would  not  suffer  from  the  obvious 
structural  deficiency  of  lacking  the  most  important  predictor.  The 
rating  from  this  model  would  be  the  expected  rating  that  ary  application 
with  that  particular  set  of  predictors  would  normally  receive  from  a 
judge.  Again,  this  could  be  used  in  two  modes,  if  the  judge  finds  no 
extenuating  circumstances  ir  the  application,  he  can  base  n:s  decision 
on  a fixed  threshold  value  and  the  predicted  score;  if  he  cheeses  to 


consider  other  factors  that  are  not  accounted  for  in  the  policy  model,  he 
can  use  the  predicted  score  as  a starting  point  and  base  his  decision  on 
this  score  plus  his  evaluation  of  the  extraneous  factors. 

Chronology  of  the  Project: 

The  original  impetus  for  this  research  came  from  a project 
in  an  advanced  statistical  applications  course.  Potential  applications 
for  policy  capturing  were  being  assessed  and  an  attempt  was  made  to 
model  Judge  #2's  policy  relative  to  a set  of  fictitious  loan  applications. 
This  led  to  the  discovery  of  four  aspects  of  the  problem  that  presented 
a challenge  to  defining  viable  policy  modets  for  loan  officers'  decisions. 
These  were: 

1)  The  methodology  of  ranking  decision  cases  as  was  the 
standard  procedure  in  most  past  Policy  Capturing 
studies  was  untenable  in  this  situation. 

2)  The  generation  of  fictitious  data  cases  in  a designed 
factorial  experiment  led  to  many  ridiculous  combi- 
nations of  predictors  and  detracted  from  the  credibi- 
lity of  the  process. 

3)  The  problem  of  scaling  and  ranking  of  the  predictors 
for  use  in  standard  statistical  routines  was  not  easily 
solved. 

4)  The  modeling  of  an  individual  loan  officer's  decisions 
as  a first-order  function  of  the  individual  predictors 
neither  produced  satisfactory  results  nor  was  it 
intuitively  satisfying  to  the  loan  officers  involved. 
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The  results  of  that  initial  effort  did  intrigue  the  participants  enough  to 
embark  upon  the  research  reported  herein  . The  major  activities  of  the 
project  are  now  described. 

Selection  ana  Coding  of  Predictors : 

The  initial  selection  of  pr edictor s was  suggested  by  the 
application  form  itself.  The  three  senior  loan  otticers  (Judges  1,?.,  3) 
were  each  interviewed  and  asked  how  they  defined  and  quantified  the 
data  or  each  of  the  input  blanks.  This  resulted  in  the  definitions  and 
codes  shown  in  Exhibit  2,  Appendix  11. 

Retrieval  of  Data : 

The  category  of  loans  under  study  consists  ol'  dealer  paper 
on  durable  appliances.  The  original  data  base  was  obtained  by  randomly 
selecting  404  loan  applications  that,  had  been  processed  by  the  loan 
officers  during  the  period  from  May  to  Si  ptember , 1971.  l.oan  appli- 
cations from  two  dealers  were  drawn  from  the  tiles  and  coded  according 
to  the  categories  determined  in  the  previous  step.  Those  predictors 
such  as  age,  income  and  equity  which  were  directly  tram.cr ihable  wen- 
coded  by  the  investigator . Those  predictors  that  required  home  judg- 
ment in  themselves  were  referred  to  the  loan  officer  who  bad  originally 
processed  the  application  and  lie  performed  the  coding. 

Analysis  and  Ordering  of  Predictor  Ca  egor ics: 

Initial  analyses  were  performed  on  a set  of  404  coded 
3 

In  this  respect  the  exercise  of  expert  judgment  in  tlu-  measurement 
function  as  advocated  by  Sawyer  (I960)  and  Minhorn  (1972),  (See 
Chapter  II)  was  inherently  accomplished  as  part  of  the  procedure. 


applications.  For  those  variables  that  were  naturally  monotonic  such  as 
age  and  income,  AID  analyses  were  used  to  determine  at  which  break- 
points, category  definitions  were  meaningful.  For  those  variables  which 
were  not  naturally  ordered,  the  categories  were  assigned  ranks  based 
upon  the  average  approval  rate  for  all  applications  in  that  category. 

This  procedure  is  discussed  in  sections  3.  ?..  4 and  3.  2.  6.  2 of  Andcrbcrg 
(I ->71).  At  this  [>mnt  n<>  .>  'iMimptions  were  made  as  to  the  interval  nature 
of  the  dj.ta  and  the  predictor  values  were  considered  as  ordered  cate 
gov i vi  only. 

.Initial  Analysis  III  for  is  : 

The  categorized  data  from  the  sample  applications  was  used 
in  an  initial  aeries  ol  computer  runs  that  were  directed  primarily  at 
answering  the  following  lour  questions: 

1)  Can  predictor  variables  and  decisions  from  historical 
filer  be  used  to  capture  the  judge's  policy  without 
requiring  him  to  make  new  decisions  on  different 
cases? 

2)  Can  the  historical  data  taken  from  the  file  be  used 
tc  build  a model  that  adequately  predict;;  the  actual 

a jipro va  1 /denia  1 .s  without  considci  ing  Credit  Hating? 

3)  What  is  the  structure  of  the  decision  process  and 
can  the  AIL)  Tree  adequately  depict  this  structure? 

4)  Cun  the  AID- Program  he  used  to  detect,  and  isolate 
inter-judge  policy  differences? 

With  respect  to  question  two,  the  reason  (or  the  exclusion  of  Credit 
Haling  lies  in  the  fad  that  Credit  Hating  is  the  only  predictor  that  la 


not  avail- iblo  immediately  after  completion  of  the  application  and 
requires  an  input  from  a source  external  to  the  dealer-lender  subsystem. 
(See  f igure  S-  1 , ) 

Definition  of  a New  Predictor : 

A major  result  of  these  initial  efforts  was  the  classification 
and  delineation  of  the  concept  of  Financial  Reference.  Originally  this 
predictor  had  been  stall'd  as  Credit  Habit  and  was  given  the  subjective 
categories  of  good  and  had.  The  initial  analysis  indicated  that  it  was 
the  second  most  important  predictor.  Its  importance  led  to  a discussion 
of  what  constituted  a good  or  bad  Credit  Habit  in  the  opinion  of  the  senior 
loan  officers.  After  some  debate,  these  officers  agreed  that  the  concept 
would  be  better  labeled  as  Financial  References  and  was  a measure  of 
the  depth  arid  quality  of  the  applicant’s  borrowing  habits.  The  specific 
categories  they  defined  were: 

Major  - - the  credit  references  listed  by  the  applicant  include 
credit  depth  on  experience  with  major  lending  and  financial  institutions 
which  do  not  lake  undue  credit  risks  and  from  whom  a reliable  credit 
refer I'uci  might  he  obtained.  This  category  includes  banks,  savings 
and  loan*;  associations,  major  chain  department  stores,  and  national 
based  lending  agencies. 

Minor  - - the  credit  references  listed  by  the  applicant  include 
only  local  stores  and  usinesscs  from  whom  a direct  credit  check  was 
not  readily  available  or  necessarily  reliable.  The  credit  references 
do  not  indicate  significant  depth  of  credit,  i.  e.  , experience  of  the 
applicant  in  managing  his  t nances  to  meet  a continuing  repayment  of 
a loan  of  size  comparable  to  that  being  applied  for.  Example  s of 
applii. ints  in  this  category  would  be  those  listing  only  small  revolving 
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charge  accounts,  such  as  oil  companies  or  those  listing  only  local  concerns 
which  carry  their  own  short-term  credit. 

Other- -the  credit  references  listed  by  the  applicant  indicated 
reliance  upon  lending  institutions  specializing  in  high-risk  high- interest 
loans.  The  references  indicated  above  average  dependence  upon  borrowed 
money  or  lack  of  capability  to  manage  one's  money  effectively  to  meet 
normal  living  expenses.  Those  people  listing  no  references  at  all  were 
included  in  this  category. 

Refinement  of  the  Sample  Data  Set : 

A second  refinement  of  the  data  set  involved  the  concentration 
of  effort  on  modeling  the  policy  of  the  judges  toward  only  one  of  the  dealers, 
namely,  Dealer  A.  This  resulted  in  the  reduction  of  the  data  set  to  224 
cases,  of  which  100  had  been  approved  and  124  had  been  denied.  These 
data  are  shown  in  Exhibit  4,  Appendix  B. 

Generation  of  Codified- Predictor  Data  and  Collection  of  Judges'  Decisions: 

Using  the  data  entries  for  each  of  the  224  cases  (See  Exhibit 
4,  ),  codified  predictor  data  for  each  application  was  generated  in  terms 
of  the  descriptors  for  each  of  the  predictor  categories.  Ihe  predictor 
descriptors  for  each  of  the  cases  were  displayed  on  a computer  printout 
(Exhibit  5,  Appendix  B)  and  used  by  the  judges  to  make  new  decisions 
for  each  application.  The  judges  were  given  one  set  of  codified  predic- 
tor data  which  included  all  of  the  predictors  except  Credit  Rating 
and  were  asked  to  make  their  decision  in  the  context  of  "tentative 
approval".  Eater,  they  ware  given  a second  set  of  data  which  included 
the  Credit  Rating  and  were  asked  to  make  the  decision  in  the  context 
of  granting  final  "approval"  or  "denial". 
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As  part  of  this  activity,  the  two  junior  loan  officers,  #4 
and  # 5,  wore  brought  into  the  project.  They  were  informed  of  what  had 
previously  transpired  and  the  objectives  of  the  experiment;  each  of  the 
three  senior  loan  officers  discussed  his  policy  as  depicted  by  the  AED- 
Tree.  The  discussion  that  followed  was  possibly  one  of  the  more 
enlightening  aspects  of  the  entire  project.  The  major  points  of  this 
discussion  revealed  the  following  situation: 

1)  Since  the  installment  department  was  in  a state  of 
rapid  expansion,  new  people  were  hired  and  put  to  work  immediately 
without  benefit  of  any  formal  training  program.  Loan  officers  were 
generally  assigned  to  collection  activities  initially  and  later  allowed  to 
evaluate  applications  under  the  supervision  of  the  senior  loan  officers. 
However,  in  this  latter  phase  the  supervision  generally  took  the  form 
of  getting  a senior  loan  officer's  approval  on  the  final  decision  about  an 
application.  Although  this  procedure  docs  provide  some  feedback  in  the 
cases  where  the  junior  and  senior  loan  officers  reach  a different  decision, 
it  apparently  does  not  suffice  to  provide  the  junior  loan  officers  with  a 
good  understanding  of  how  the  senior  loan  officers  evaluated  each  of 

the  entries  on  the  application  form.  Thus,  the  discussion  of  the  appli- 
cations to  be  rated  appeared  to  provide  the  catalyst  for  the  first  compre- 
hensive exchange  of  opinion  among  all  five  of  these  loan  officers. 

2)  The  concept  of  Financial  Reference  as  previously 
agreed  upon  by  the  three  senior  loan  officers  was  not  a part  of  the 
policy  of  the  junior  loan  officers. 

Generation  of  Policy  Models : 

Upon  completion  of  the  evaluations  by  each  of  the  loan 
officers,  policy  models  for  each  of  the  officers  were  developed  using 
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th  ■ techniques  discussed  in  Chapter  IV.  These  models  arc  presented 
and  discussed  in  Chapter  VI. 

At  this  point  a different  problem  became  apparent  in 
using  these  models  as  a predictive  tool.  The  problem  is  that  of  which 
loan  officer's  model  to  use  on  which  application.  In  the  current  opera- 
ting scheme,  the  incoming  applications  generally  are  not  routed  to 
any  particular  individual,  but  are  acted  upon  by  whomever  is  available. 
This  factor  not  only  posed  a problem  for  adequately  evaluating  the 
performance  of  a given  loan  officer's  model  but  also  has  the  effect  of 
making  an  applicant's  chances  for  getting  a loan  a function  of  which 
judge  evaluates  it,  and  what  his  idiosyncrasies  may  be.  Therefore, 
several  voting  strategies  were  investigated.  The  strategy  that  best 
matched  the  actual  decisions  was  determined  to  be  one  in  which  a loan 
was  approved  if  four  of  the  five  of  the  individual  loan  officers  approved 
it. 


Validation  by  Field-Test: 

Two  of  the  original  goals  of  this  research  were  to  analyze 
the  problems  that  might  be  encountered  in  the  implementation  of  these 
policy  models  in  the  real  en vironment  and  to  demonstrate  the  adequacy 
of  the  models  by  cross-validation.  To  this  end,  the  appliance  dealer 
who  was  the  source  of  the  ZZ*1  applications  that  were  used  in  the 
modeling  phase  was  approached  and  asked  to  participate  in  a one- 
month  field-test.  This  field-test  took  the  form  of  having  the  dealer's 

A 1 /,  / f 1 1 «•<.  a « 4>  4-  U a /l  nr)  «s  •»%  J « n fa  o f aA  *•  *■>■>  Hi'  J K if  A A »"i  n **»  ^ 
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B,  in  addition  to  the  normal  application  form  on  each  sale  that  required 
financing  during  July,  197Z.  In  order  to  accomplish  this  task,  the 
clerical  staff  was  briefed  on  the  project  and  given  a set  of  instructions 
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as  shown  in  Exhibit  6,  Appendix  B.  Upon  receipt  of  the  original 
application,  the  loan  officers  also  filled  oat  the  coded  form  but  made 
their  decision  in  the  same  manner  they  had  always  done.  Daring  the 
month,  150  usable  applications  were  collected  which  were  then  analyzed 
in  several  ways.  These  analyses  included: 

1)  evaluation  of  model  sensitivity  coding  differences 
between  "exper'  measurement"  of  the  loan  officers 
and  the  "non-expert  measurement"  of  the  clerical 
staff, 

2)  cross-validation  of  the  previously  derived  individual 
and  voting  policy  models, 

3)  evaluation  of  the  average  time  and  range  of  times  the 
applications  were  in  process. 

An  Insight  into  Transferability: 

One  of  the  interesting  findings  arising  out  of  the  review  of 
past  experience  with  numerical  credit  scoring  systems  is  the  fact  that 
such  systems  have  been  "borrowed"  as  opposed  to  "tailored"  for  new 
enviornment  A natural  question  then  is  "are  policy  models  captured 
in  one  environment  amenable  to  being  transferred  to  new  environments?" 
Although  an  adequate  answer  to  this  question  would  require  a major 
research  project  itself,  an  attempt  was  made  to  get  an  indication  of 
the  potential  transfer rability  of  the  models.  In  this  effort  two  experi- 
enced loan  officers  who  were  managers  of  other  lending  agencies  in 
Austin  were  interviewed  and  asked  to  make  decisions  based  on  the  co- 
dified predictor  on  the  224  applieat  one.  One  of  these  individuals, 

Judge  #6  had  recently  been  hired  by  the  participating  installment  credit 
department  but  had  not  yet  worked  with  them  long  enough  for  their 
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policies  to  influence  his  opinion. 

Summary: 

This  chapter  has  briefly  reviewed  the  past  work  in 
numerical  credit  scoring  and  has  described  the  environment  in  which 
this  experiment  was  conducted. 

A short  chronological  description  of  the  research  activities 
relative  to  capturing  and  evaluating  the  policy  models  for  the  loan 
officers  has  been  given.  The  various  forms  and  instructions  generated 
and  used  in  these  activities  nave  been  included  in  Appendix  B.  The 
results,  conclusion  and  implications  of  this  project  are  discussed  in 
Chapter  VI. 
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CHAPTER  VI:  ANALYSIS,  RESULTS,  AND  IMPLICATIONS 

OF  LOAN  OFFICER  POLICY  MODELS 


Introduction: 


In  documenting  the  results  of  this  research,  it  might  be 
sufficient  to  simply  list  the  final  set  of  equations  and  their  cross-vali- 
dation hit  rates,  however,  many  initial  and  intermediate  results 
were  obtained,  each  of  which  influenced  the  direction  of  the  research 
activities  that  followed  to  some  extent.  Not  all  of  the  results  were 
positive,  but  even  those  that  were  negative  added  to  the  knowledge  of 
the  investigator.  It  would  seem  somewhat  pretentious  not  to  discuss  all 
of  these  results  for  two  reasons;  a)  they  may  answer  questions  of 
later  investigators  and  help  them  avoid  unnecessary  stops,  or  b)  they 
may  provide  the  impetus  fur  better  ideas  or  decisions  than  those  which 
were  pursued  by  this  investigator.  Therefore,  this  chapter  includes 
the  intermediate  results  as  well  as  the  final  equations  in  the  hope  of 
providing  a better  insight  into  what  worked  and  what  did  not  work. 

The  breadth  of  the  results  presented  in  this  chapter  may  create  a 
problem  in  assimilating  the  information  presented.  To  alleviate  this 
problem,  the  presentation  has  been  further  segmented  into  topical 
sections  and  the  following  directory  is  provided. 


Section  Topic  Pages 

I Results  and  Decisions  of  Initial  Analysis  116 

II  Analysis  of  Decisions  for  Codified  Predictor 

Sets  126 

III  Presentation  of  Individual  Policy  Models  134 
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163 
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167 
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180 
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Section  1. 


Analysis  and  Refinement  of  the  Original  Sample: 


The  original  sample  of  data  included  404  applications  .vhich 
were  randomly  selected  from  the  files  of  the  lending  institution.  Of 
these,  270  applications  had  been  submitted  by  a high  volume  dealer 
(Dealer  A)  who  had  been  participating  in  a dealer  contract  arrangement 
with  the  lender  for  a period  of  approximately  six  months.  The  other 
1 34  applications  were  from  a lower  volume  dealer  (Dealer  B)  who  had 
been  working  on  a dealer  contract  basis  with  the  lending  inr ' itution 
for  several  years. 

The  data  on  each  application  was  coded  in  terms  of  one  of 
the  category  descriptors  shown  in  Exhibit  2,  Appendix  B.  For  those 
predictors  that  were  not  naturally  monotonic,  the  categories  were 
rank  ordered  according  to  the  average  success  rates  for  all  applica- 
tions within  each  category.  The  data  in  Table  6-1  reflects  the  success, 
or  approval,  rate  for  each  category  of  each  predictor  and  was  the  basis 
for  the  category  rankings. 

TABLE  6-1 

BY- CATEGORY  SUCCESS  RATES  USED  FOR  RANKING 
OF  PREDICTOR  CATEGORIES 
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Variable  \ Category 

1 

2 

3 

4 

Ago 

. 50 

. 57 

. 58 

. 79 

Marital 

. 59 

. 63 

. 63 

Local  Family 

. 58 

. 6l 

Draft 

o 

o 

. 60 

Telephone 

. 35 

. 63 

Average  Time 

. 53 

. 62 

. 68 

Residence  Cat, 

, 25 

. 51 

. 72 

00 

00 

Employ.  Type 

oo 

. 62 

. 64 

. 65 

. 72 
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Table  6- 1 (cont.  ) 


Variable  \ Category 

1 

2 

3 

4 

5 

6 

Income 

. 44 

. 49 

. 64 

. 48 

. 59 

.74 

Bank 

. 33 

. 67 

Credit  Habit 

. 12 

. 70 

Loan  Amount 

. 53 

. 59 

. 56 

. 66 

. 60 

. 70 

Loan  Term 

. 68 

. 53 

. 55 

. 57 

Neccs  sity 

. 54 

. 69 

Equity 

. 56 

. 63 

. 74 

Credit  Rating 

. 10 

. 55 

, 65 

. 94 

The  initial  analysis  of  the  data  set  using  both  bi- variate 
contingency  tables  generated  with  Anderber g' s (1971)  GCORR  program 
and  output  from  the  A1D4  program  led  to  several  refinements  and  modi- 
fications to  the  predictor  set.  These  included: 

1)  Elimination  of  draft  as  a variable  since  draft  eligi- 
bility automatically  eliminated  an  applicant  from  con- 
sideration based  on  company  policy. 

2)  Elimination  of  debt  as  a predictor  since  useable  data 
was  available  on  less  than  1/3  of  the  applications. 

3)  Reduction  in  the  number  of  categories  required  to 
describe  the  loan  amount  and  income  predictors. 

4)  Inclusion  of  self-employed  and  military  officers 
into  the  executive  category  based  on  indifference  of 
initial  AID  analyses  to  their  classification  in  separate 
categories. 

5)  Elimination  of  those  applicants  achieving  loans 
through  "full  recourse"  guarantees  of  the  dealer. 
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These  modifications  resulted  in  a refined  data  set  of  389  applications, 
262  from  Dealer  A,  and  127  from  Dealer  B, 


Initial  Analysis  Disregarding  Credit  Rating  as  a Piedictor: 


The  389  data  cases  were  used  in  exploratory  efforts  to 
build  a model  that  would  adequately  predict  the  actual  response  without 
the  use  of  credit  rating  as  a predictor.  Various  modeling  alternatives 
including  multiple  regression,  multiple  classification  analysis,  and 
AID-Trees  were  pursued. 

These  efforts  were  generally  unsuccessful,  resulting  in 
first-order  and  second-order  equations  that  could  account  for  only  20- 
25%  of  the  variability  and  AID-Trees  that  required  6i  splits  (62  mutually 
exclusive  groups)  to  account  for  60%  of  the  variability.  ^ The  primary 
reason  for  the  poor  quality  of  these  early  models  was  .assessed  to  be 
structural  deficiency  in  the  models.  The  actual  decisions  had  been 
made  considering  the  Credit  Rating  of  the  applicant.  Attempts  to 
model  these  actual  decisions  without  using  Credit  Hating  as  a predictor 
represented  omission  of  the  single  most  important  variable  in  the 
judgment  process. 

A second  task  of  the  preliminary  analysis  was  to  determine 
if  a better  fitting  model  (in  terms  of  unexplained  variability)  migl  t be 
obtained  by  using  a continuous  criterion.  To  this  end,  the  loan 
officers  had  been  asked  to  rate  each  of  the  applications  on  a scale 

* Throughout  the  initial  analyses,  discussed  in  Section  I,  Chapter  VI, 
the  square  of  the  multiple  correlation  coefficient  (U^)  was  used  as 
the  criterion  of  goodness  for  making  comparisons  and  decisions. 

Bater  the  hit  rate  (see  Chapter  IV,  page  84  ) was  adapted  as  a morn 
appropriate  criterion  and  used  for  the  remainder  of  the  research. 
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of  1 to  5.  Figure  6-1  shows  the  comparison  of  the  variability  explained 
as  a function  of  the  number  of  mutually  exclusive  groups  for  both  the 
binary  and  scaled  criteria.  Only  Judge  1 and  Judge  3 had  processed  a 
sufficient  number  of  the  original  applications  to  perform  the  analysis,  but 
in  both  cases  their  scaled  criterion  values  were  less  predictable  than 
their  binary  decisions.  Review  of  the  residuals  for  each  of  these 
approaches  revealed  that  most  of  the  added  variance  experienced  in  the 
scaled  rating  models  was  attributable  to  those  applications  that  were 
denied,  possibly  indicating  a lower  thresholding  effect  in  the  judges' 
policies,  i.  e.  , a good  application  had  measurable  levels  of  "goodness" 
but  a bad  application  was  just  "bad".  These  results  would  suggest  that 
having  the  judge  rate  or  rank  the  data  cases  in  a manner  different  from 
their  natural  decision  process  would  tend  to  produce  erroneous  policies 
models. 

Another  exploratory  approach  involved  attempting  to  define 
"factors"  entering  into  the  judgment.  In  this  effort  Factor  Analysis 
computer  routines  were  used  to  combine  13  predictors  into  five 
"factors".  These  five  factors  were  capable  of  explaining  some  60% 
of  the  internal  variability  within  the  predictor  set.  However,  the 
resulting  combinations  of  "ariables  in  each  of  the  factors  were  not 
particularly  meaningful  to  the  loan  officers  who  stated  that  they  really 
thought  of  loan  applications  interrnsof  the  following  four  factors: 

1)  Capacity:  capability  to  service  debt 

2)  Character:  stability  of  applicant 

3)  Credit:  past  credit  record 

4)  Condition:  particulars  of  loan  and  risk  to  lender. 


NUMBER  OF  MUTUALLY  EXCLUSIVE  GROUPS 


FIGURE  6-1 
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Since  one  oi  the  underlying  objectives  of  the  research  project  was  to 
model  known  policies  ot  the  loan  officers  in  meaningful  terms,  the 
"factors"  that  were  mathematically  defined  were  discarded  and  not 
considered  further. 

i he  analysis  and  comparison  ol  the  AID- Trees  for  each  of 
the  dealers,  on  an  individual  basis,  did  result  in  illumination  of  an 
important  point,  Tin-  trees  showed  an  apparent  difference  in  the  policy 
ol  the  lending  institution  toward  each  of  the  dealers.  In  the  case  of 
Dealer  A,  only  h 2 “ii  of  the  applications  were  appioved  while  7S%  of 
the  applications  from  Dealer  It  were  approved.  Further,  AID- Trees 
reflected  that  different  variables  were  important  in  explaining  the 
variability  in  each  ol  the  policies.  For  Dealer  A,  the  most  important 
predictors,  as  indicated  Ly  the  first  iew  splitN  of  the  AID-Tree,  include 
Bank  Account,  Age,  Fquily,  and  income,  For  Dealer  11,  the  most 
important  predictor  s were  Residence,  F.mploy  inent  Type,  Loan  Term, 
and  Local  1'u.mily.  This  apparent  difference  was  discussed  with  the 
judges  and  they  rellccled  that  there  actually  was  a difference  in  their 
attitude  toward  the  different  dealers,  they  noted  that  this  difference 
was  nut  explicit  but  was  implicit  and  arose  from  ihuir  working  relation- 
ship will  the  individual  dealers  and  the  types  of  applications  they  re- 
ceived from  each.  Specific  causes  of  the  differences  were: 

1)  Dealer  11  has  a more  affluent  and  mobile  clientele 
than  did  Dealer  A. 

2)  They  had  been  operating  with  Dealer  11  for  a much 
longer  time  and  they  expected  dial  some  pre-screening 
of  the  applications,  based  on  their  accept /rejec  t ex- 
perience, occurred  before  flu  applications  were 
forwarded . 
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3)  Dealer  A was  more  helpful  in  recovery  and  disposal  of 
repossessed  merchandise  and  so  they  were  relatively 
more  interested  in  the  variables  that  reflected  the 
applicant's  "capability"  to  pay  than  they  were  in  those 
reflecting  his  "credit"  or  "character". 


Preliminary  analysis  for  each  of  the  three  senior  loan 
olficers  did  indicate  difference-1'  in  the  variables  each  considered  the 
most  important  and  served  to  convince  them  that  the  AID- Trees  were 
a viable  means  of  portraying  and  analyzing  their  policies.  For  example, 
Figure  6-2  reflects  the  success  rates  for  each  of  the  categories  of  the 
predictor  Marital  Status  for  the  three  senior  loan  officers,  It  indicates 
that  being  single  is  somewhat  of  a detriment  in  the  eyes  of  Judge  1 whiie 
being  divorced  is  a detriment  from  Judge  3's  viewpoint.  It  is  interesting 
to  note  that  during  the  formulation  of  the  predictor  categories,  all 
three  judges  had  expressed  the  opinion  that  a married  applicant  had  a 
better  chance  of  getting  a loan,  but  their  responses  reflected  this  to  be 
true  only  in  the  case  of  Judge  1.  However,  a cautionary  note  is  in  order 
here!  The  assertion  of  a given  attribute  being  "belter"  should  be 
interpreted  in  the  context:;  of  "all  other  things  being  equal".  Hence, 
the  zero-order  correlation  between  "married"  and  "approval"  may 
not  reflect  the  true  importance  of  the  marital  status  predictor  in  the 
situation  where  there  are  other  intervening  predictors  that  are  not 
independent  of  marital  status.  Nevertheless,  the  use  of  plots  such 
uo  Figure.  6-2  was  found  to  be  a good  catalyst  for  discussion  of  the 
individual  loan  officer's  policies. 

Although  differences  between  judges  and  dealers  were 
delectable  in  the  early  analysis,  the  inability  to  use  only  historical 
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decisions  to  obtain  models  that  were  sufficiently  predictive  caused  con- 
siderable consternation  and  re-analysis  of  the  decision  process  and 
modeling  methodology.  Three  basic  problems  were  identified: 

1)  The  transformation  from  an  AJD-Tree  into  a configural 
effects  regression  model  requires  a different  conceptu- 
alization of  interaction  among  the  predictors  than  is 
indicated  by  the  standard  interaction  term  of  the  form 

b12*lV 

2)  Decision  processes  probably  cannot  be  modeled  very 
successfully  if  all  of  the  important  variables  that 
entered  into  the  original  decisions  are  not  included  in 
the  model.  For  the  loan  officers,  there  is  no  way  to 
partial  out  the  effect  of  Credit  Rating  from. the  actual 
responses  that  had  been  made  with  it  being  considered. 

3)  In  situations  where  the  number  of  data  points  is  rela- 
tively small  compared  to  the  number  of  predictor 
categories  (389/53  in  this  data),  uncontrolled  variation 
in  the  input  data  used  by  the  judges  must  be  minimized 
to  improve  the  "signal"  to  "noise"  relationship. 


The  solution  to  these  three  problems  was  pursued  in  two 
directions,  the  first  being  the  methodological  development  as  discussed 
previously  in  Chapter  IV;  the  second  being  the  generation  of  the  coded 
predictor  data  and  the  re-evaluation  of  this  data  by  the  three  original 
senior  loan  officers  and  two  junior  loan  officcx  .■>  \ j it4  auu  uUuge  tt5/, 
The  re-evaluation  of  the  applications  based  on  the  predic- 
tor "descriptors"  was  undertaken  for  two  purposes: 

1)  to  eliminate  from  consideration  any  extraneous 
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2) 


information  that  had  appeared  on  the  original 
application  and  thereby  assess  the  effect  of  controlling 
the  information  on  the  application  form,  and, 
to  facilitate  interjudge  comparisons  by  allowing  com- 
parison of  each  loan  officer's  opinion  on  each  loan. 


This  procedure  reflects  adherence  to  Hoffman's  (I960)  prim  iple  of 
experimental  control.  Although  the  raw  historical  data  did  not  permit 
definition  of  adequate  policy  models,  it  did  provide  a judgment  data  set 
reflective  of  the  population  of  actual  decision  cases  that  are  encountered 
in  the  real  world. 

Prior  to  regeneration  of  the  predictor  data,  further  refine- 
ment of  the  predictors  was  accomplished  with  the  final  set  of  predictors 
consisting  of  the  16  variables  shown  in  Exhibit  3,  Appendix  B.  The 
refinement  was  accomplished  on  the  applications  from  Dealer  A.  Of 
the  original  262  applications,  only  224  could  be  retrieved  from  the 
installment  company's  files  due  to  the  fact  that  the  files  had  been  relo- 
cated and  reorganized  and  some  accounts  had  been  closed  out  during 
the  period  of  the  initial  analysis.  This  situation  in  itself  reflected 
one  of  the  difficulties  of  performing  a Policy  Capturing  analysis  with  the 
use  of  actual  data  from  a functioning  organization.  In  order  to  cause 
a minimal  disturbance  on  the  on-going  system,  the  original  applications 
had  been  left  in  the  system  and  were  not  totally  accessible  at  a later 
date. 
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Section  II. 

Decisions  Based  on  the  Codified  Data  Set:  (Without  Credit  Rating) 

Each  of  the  judges  was  given  a set  of  the  Codified  Predictor 
Sheets  with  all  predic  ors  except  Credit  Rating  and  asked  to  either  grant 
"tentative  approval"  or  "reserve  judgment".  During  this  process,  con- 
siderable discussion  resulted  relative  to  which  predictors  were  impor- 
tant and  which  were  not  important.  This  was  one  of  the  particularly 
interesting  and  beneficial  aspects  of  the  study  in  that  it  represented 
the  first  time  all  of  the  five  loan  officers  had  extensively  discussed 
their  philosophies  and  policies  relative  to  evaluating  loans. 

Since  the  objective  of  the  decisions  made  without  benefit  of 
the  Credit  Rating  was  to  identify  only  the  best  prospects  for  "tentative 
approval",  the  loan  officers  defined  seven  conditions,  any  one  of  which 
would  automatically  prohibit  tentative  approval  regardless  of  the  overall 
quality  of  the  application.  These  conditions  were: 

1)  under  21  years  of  age 

2)  under  2 years  residence  time  and  under  2 years  job 
time 

3)  over  5 years  residence  time  and  still  renting 

4)  over  25  years  old  without  major  credit  rcPr^nces 

5)  under  $400  income 

6)  no  bank  account 

7)  "other"  financial  reference  (code  1) 

These  conditions  reflect  a conservative  bias  on  the  part 
of  the  judges  since  they  were  viewed  as  significant  weaknesses  in  the 
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applications  that  made  a positive  decision,  in  the  absence  of  a credit 
rating,  unduly  risky.  These  rules  were  implemented  in  the  models 
by  defining  a binary  predictor  that  indicated  presence  of  one  or  more  of 
the  seven  conditions,  for  want  of  a better  term,  this  predictor  was 
labeled  "Dings".  Use  of  Dings  as  a predictor  allowed  much  better  fitting 
models  for  each  judge's  "without  credit  rating  decisions".  However, 
the  match  between  the  decisions  predicted  by  the  models  and  the  actual 
approvals /denials  was  still  not  as  good  as  desired. 

There  was  considerable  variation  in  which  loans  the  judges 
individually  gave  "tentative  approval"  and  only  66%  of  the  224  decisions 
were  unanimous.  More  importantly,  only  20%  of  the  applications 
received  unanimous  "reserve-judgment- without- credit"  decisions.  This 
is  partly  attributable  to  denying  the  judges  use  of  their  primary  predic- 
tor and  partly  due  to  the  fact  that,  in  making  the  "tentative  approval" 
decision,  they  were  really  making  a different  decision  than  they  were 
normally  accustomed  to  making.  In  an  effort  to  determine  if  a compo- 
site evaluation  might  be  better  for  isolating  just  the  very  best  prospects, 
various  voting  strategies  were  investigated.  It  was  found  that  a strategy 
based  on  granting  tentative  approval  if  four  of  the  five  judges  had  indi- 
vidually given  "tentative  approval"  to  the  application  and  did  the  best 
job  of  identifying  the  top  25-30%  of  the  applications  while  minimizing  the 
inclusion  of  applications  which  had  actually  been  denied.  The  criterion 
of  25-30%  was  arrived  at  based  on  discussion  with  loan  officers  relative 
to  what  nnrrnntauR  of  "tentative  aDDrovals"  would  be  reouireri  to  make 
a policy  model  beneficial  to  them.  Voting  models  considering  the 
"Dings"  reflected  some  over  - conser  vatism  in  that  only  71  out  of  224 
cases  would  have  received  tentative  approval  and  11  of  these  71  had 
actually  been  denied.  Voting  models  that  did  not  use  "Dings"  as  a 
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predictor  performed  approximately  the  same  since  74  of  the  224  cases 
would  have  been  tentatively  approved  with  12  mistakes.  The  regression 
coefficients  for  those  models  are  presented  in  Tabic  6-2. 

Again,  one  of  the  primary  results  of  this  effort  came  from 
discussion  and  clarification,  in  the  judges’  minds,  of  which  attributes 
were  indicative  of  abnormal  risk. 

Decisions  Based  on  Codified  Data  Set:  (With  Credit  Rating) 

Each  judge  was  given  Codified  Predictor  Sheets  which 
included  the  Credit  Rating  for  each  of  the  applicants.  They  were  askud 
to  make  their  decisions  as  if  they  were  final  approvals  or  denials  on  the 
loan.  This  set  of  decisions  provided  the  data  for  comparison  of  the  five 
judges  on  an  inter-judge  basis.  Decisions  on  the  Codified  Predictor 
Sheets  were  made  twice  by  each  judge,  thus  providing  information  on 
the  reliability  (percent  of  decisions  that  were  the  same  on  both  trials) 
for  each  judge.  Comparison  of  the  decisions  made  on  each  set  of 
codified  predictor  data  with  the  actual  decisions,  that  had  been  made 
previously  from  the  original  application  form,  provided  insight  into 
the  amount  of  "information"  lost  in  the  process  of  codifying  the  predic- 
tors. Tabic  6-3  contains  the  hit- table  for  the  coded  versus  actual 
decisions  and  the  reliability  of  each  of  the  judges.  Since  each  of  the 
original  applications  had  been  evaluated  by  only  one  of  the  three  senior 
judges,  the  matches  between  the  original  decisions  of  each  judge  and 
hvs  decisions  on  the  corresponding  sets  of  codified  predictors  is  also 
shown. 
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TABLE  6-2 

TENTATIVE  APPROVAL  MODELS 
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PREDICTED  SCORE-CONSTANT  + £ COEFFICIENT  x PREDICTOR 

1 i i 


Predictor:  Coefficient 


1. 

Constant 

-1. 377 

2. 

Age 

- .0125 

3. 

Mar  ital 

- .0542 

4. 

Local 

. 0355 

5. 

Phone 

. 231 1 

6. 

J obtime 

. 0536 

7. 

Restime 

. 0387 

8. 

Reseat 

. 0918 

9. 

Typo  Employment 

. 0230 

10. 

Income 

. 0164 

11. 

Bank  Account 

. 1200 

12. 

Financial  Reference 

. 3002 

13. 

Loan  Amount 

. 0328 

14. 

Luan  Term 

- .0352 

15. 

Necessity 

. 0371 

16. 

Equity 

. 0275 

17. 

Dings 

- . 1200 

* 


Model  without  dings  includes  only  first  16  terms. 
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The  key  point  derived  from  the  comparison  of  the  judges' 
decisions  made  on  the  codified  predictors  with  their  original  decisions 
based  on  the  full  application  is  that  codifying  the  data  only  slightly 
affected  the  decisions.  This  assertion  is  made  since  the  reliability 
of  each  of  the  three  senior  judges  was  approximately  90  percent  and 
the  correspondence  of  their  decisions  based  on  the  two  forms  of  data 
ranged  between  85  and  87.  5 percent. 

The  inclusion  of  the  Credit  Rating  predictor  and  the  refine- 
ment of  the  categories  of  the  other  predictors  resulted  in  attainment  of 
a vastly  improved  fit  to  the  data.  Figure  6-3  shows  the  AID-Tree 

2 

developed  using  the  actual  decisions  as  the  criterion.  In  the  total  tree  , 
nine  splits  occurred  leaving  ten  mutually  exclusive  groups  which  accounted 
for  "6  percent  of  the  variability  in  the  data. 

Table  6-4  shows  the  ranked  importance  for  the  variables 
re’ative  to  their  power  to  explain  variability  over  the  total  predictor 
ace  and  also  over  the  predictor  subupaces  defined  by  the  AID  splitting 
process.  Tiie  individual  predictors  are  ranked  according  to  their 
average  BSS/TSST  and  13SS/TSS(i)  values.  The  data  in  this  table  would 
indicate  that  CREDIT  RATING,  FINANCIAL  REFERENCES,  BANK 
ACCOUNT,  LOAN  TERM,  TYPE  EMPLOYMENT  and  PHONE  are  the 
most  important  variables  in  the  decision  process.  Since  they  hold 
ranks  1 through  6 in  both  rankings,  they  are  important  for  all  applicants. 
MARITAL  STATUS,  LOCAL  FAMILY,  and  NECESSITY  hold  very  low 
ranks  in  both  rankings  and  are  the  least  important  predictors. 

2 

Only  eight  of  the  splits  actually  appear  on  Figure  6-3  due  to  limitations 
in  this  version  of  the  tree,  printout.  The  expanded  version  of  the  AID- 
Tree  shown  in  later  examples  can  portray  12  levels  of  the  splitting 
process . 


JRE  6- 


>-uiiMn 


1 


1 33 


coo  uooooucccoc  oc 


roi/W)> 
i'oNn  o f"  pi  .»  » rj  r-  (?  t\j  ♦ 
« iP(P  — ip  -*o  w < pi  — or) 


1 ^ NO  iv  IT>^«  w — #tv» 

DOOfUDOHOumnfMO 

JOQOOOOOOU^-OOO 

i o oo  o o o o o ooo  ooo 

r O O O OOOOOOOOQOO 


f»*  w 

own 

otr 


c o oo  n e 

► P»  t>  ~o  fu  j< 


70000  OOOOO 
4 ® O (V  « «.  "i  4 0 — 


*<\jPjP'fwt\t-*'PgP 


V-<UNO 


*-  p-  o >M  r-  r-  or^s  ^ oi/\«» 

j W *0  PI  -9  » S — • « **1  M *-  * 4>  -O  T> 
1 ♦OKM")'7^-ON*piMO^ 
O O in 


«<«gv-»upiw®r>  ♦-.-.-inf-* 
ki  n oo^o  o o^oin^N  • oo® 
ut/oooooooooooooo-i 
7000000000000000a 


If 

I 

sO 

W 

J 

ffl 

C 

H 


o ^ o « < oiflVP-MMJf 

CPiOWCPP’-*-®  4 n n o on  w « 
« tsi  -w  ■*  t->  ■+  o + m *t  t*  •>*>-**•  * 
• niO  rv  -o  —•  o .»  m -~i  *-  ® <>  »■•  4 
o— n ocron  it-  o 4 ® ♦ » -*  a 


O^rfir-VWOn*  — < O'  0 9 9 9 
aj®»-iwwjw®w<«9  ®v  ® p 

0-4)«-«4^^r'00‘00  o ■#  fM  W 
o oo  om  m *i-i  o c 

WO  OCJ  o ooo  O Cl  OU  O t)  C « 


OVt 

D*-n  Wo  4>  OOO  «WK  « ««» 

acx»«/P»fvrvi®pnrx»er-r-*rgfv®r 

v <n  ra  r- 1-*  w «<vj  — p-  * rv,  —■  r~  4 -< 
® ® »-  ♦ .-.  4,  V ♦NPIWiN  • W « ' < 
rvooo  -<  o o-i  -*  o ♦ N 3 

OS • 

U 3 
hi  w 


1 rtOnHUI  f*  P»  — < ci  * V « « » « 

U1WBMM7WOOO  »►■  4>  ® « 

• 0 4 4 -w  o ■ n 


«c»  •ooa*o^-®*  o »-  ♦ 
« pj  .-•  « -.  — . r\t  *i  — r-  4 O n n n 
®c0OOt.CC0O0O®0OC-- 


O W 

3P-#NPWIt)Vn"'«V’0»niO*«** 

Ot  N.  O H»U7MT»  OC'IMU'OP'  4 in 
►•l/1<inOOP<  W N»IITr,lO«)0<  CM 
V<v>  n O Wo  «I  O tMf'OO  1\*  t n 

Z Oil  in  Pi  r+  "\  <•  ~1  O'  D * ® <M  4 ■'1 ' 

OS  


hJ  W 


— «nc*  ♦ w or 

I w w p'f'j  a>T>f\j  — 

- 

v»H  ooo  -«  O O 


— m 

Ol  t 

r>  3 

vr® 


« »J  0<1  WO®  ® #■ 
«n-«jiMftiwn  « * 
v«.-i^r-C*Wr<i'4 
OPirDDOWP^ 
— < F4  0MNO-HO  Cl  W 


NP14  WO  <PfWW«C®P~  < 9 ru 

* ® 4 ® W 4*  V <tNlP#-^WOU<0 
« t/ipiPtrt#<PW<,''i»«v.®o?\i4  —■<->•— 
m»- * o oj  4 »-• 
a \ * * » • = • • »•••••*«• 
< VI  -* 

> ® 

— to 
m x 
r?  j 

® Vi 


_i  in-  auiu 

i-  .i  u-i.<<  i y < 

«■<  Z *~  h-ct  WOV 
lit  ii  U O ui  v>  (n  «■  u Z 
<S<<JIOUJUJVZW 

< l j a m a v ■“» 


U-  X * *n  v* 


j ~o: 
> n -j 

J OCX 

L MJU 


-j  iji-6.iuuu.ti;  V»w 
l>  JUJiihU  I 14  UUP-  ®»-< 
•i  * n-t-OHOl'9  ITU"! 

t; ff  1 J •?  <5  ^ m I*,  ii  v y « « o iuJ 
o < o * ouwpjx-ooiuoa 
4 I J 6 TO  a-VW*)h._l_J/  VUl 


<N  A4«t«V  «9  Oi-NAI 


,<r 


RANKED  IMPORTANCE  OF  PREDICTORS 


Section  III. 


■'V  f. 


1 34 


The  Individual  Policy  Models: 

Using  the  responses  to  the  Codified  Predictor  Sets  as  the 
criterion,  first-order,  second-order,  and  A ID -do  rived  "local"  models 
were  developed  for  each^  of  the  five  judges  and  for  the  four  of  five  voting 
strategy. 

The  First  Or  dor  Models- 

The  coefficients  for  first-order  regression  models  of  the 

to  r m 


Predicted  Score  = Constant  + £ coefficient  x predictor. 

. i i 

i 

were  obtained  following  the  iterative  model  deter  ruination/ cross  vali- 
dation procedure  described  in  Chapter  IV.  Implicit  in  these  models  is 
the  assumption  that  the  predictor  categories  are  ordered  according  to 
the  codes  given  in  Exhibit  3,  Appendix  11  and  that  a constant  change  in 
the  Predicted  Score  will  result  from  a unit  change  in  the  level  of  a 
Predictor  Variable.  This  assumption  was  made  in  an  attempt  to  avoid 
idiosyncratic,  results  that  could  result  from  fitting  the  224  data  points 
with  a binary  regression  model  which  had  53  predictors.  No  attempt 
was  made  to  test  individual  terms  for  statistical  significance.  However, 
it  is  apparent  that  any  variable  whose  coefficient  is  smaller  in  magnitude 

3 

Second-order  models  were  only  derived  for  Judge  1,  2 and  3. 
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than  . 002  would  have  an  effect  on  the  total  score  of  less  than  . 01  (since 
the  range  oi  values  for  tb*'  variables  is  between  1 and  5)  and  should 
probably  be  ignored.  The  first  order  coefficients  fov  the  5 judges  are 
shown  in  Table  6-5.  The  best  cut  score  reflects  that  point  where  the 
maximum  hit  rate  occurs. 

The  Second- Order  Models: 

Second- order  regression  models  of  the  form 

Predicted  Score  Constant  1 £ coefficient,  x predictor. 

i 1 

+ £ £ coefficient.,  x predictor,  x predictor. 

i j V 1 J 

were  also  analyzed.  In  the  absence  of  substantive  theory  from  which 
to  hypothesize  the  terms  in  the  second-order  models,  they  were 
hypothesized  by  generating  all  possible  first-order  and  second-order 
terms  and  using  a stepwise  regression  routine  to  eliminate  those  terms 
not  statistically  significant  at  the  .01  probability  level.  Due  to  compu- 
tational limitations,  the  identification  of  potential  terms  was  done 
incrementally  with  16  first-order  and  approximately  4b  second-order 
terms  being  considered  in  each  of  three  runs.  All  terms  identified  by 
this  procedure  were  included  in  the  second-order  model  regardless  of 
their  "meaningfulness"  or  "interprelability".  The  iterative  procedure 
was  then  used  to  determine  the  coefficients  in  Table  6-6  for  the 
hypothesized  model  for  each  judge. 

AID- Split  Models : 

AID-Split  models  of  the  form 
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Predicted  Score  = £ group  mean,  x 


CM. 


where:  GM, 
x 


1 if  applicant  is  in  group  i 


0 otherwise 


were  obtained  by  analyzing  the  application  data  with  the  AID4UT  program. 

The  models  as  shown  in  Figures  6-4,  through  6-8,  represent  the  first 

5 or  6 splits  of  the  tree,  at  which  point  subspaces  (Figures  6-9  through 

6-13)  were  defined  and  local  models  were  developed.  In  Figures  6-4 

through  6-8,  the  group  means  are  denoted  by  "A."  and  the  split  variable 
" (k  , 1)"  1 

is  denoted  by  . The  subscript  "j"  represents  the  identification 

number  of  the  split  predictor  per  Exhibit  3,  Appendix  B.  The  super- 
scripts "(k,l)"  denotes  the  categories  of  the  split  predictor  that  are  in 
each  group. 

Local  Models: 

Local  models  of  the  form 


Predicted  Score  - 


where:  GM. 

x 


Constant  + £ GM.  x (Local  Model.) 

i 1 1 

1 if  applicant  is  in  subspace  i 
0 otherwise 


Local  Model  i - First  or  second-order  model  pertaining 
only  to  those  applicants  in  subspace  i. 

were  derived  based  on  analysis  of  the  AID-Trees.  In  the  process  of 
developing  the  local  models,  several  alternative  formulations  within 
the  various  subspaces  were  investigated.  The  general  result  was  that, 
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AID  SPLIT  DIAGRAM 
JUDGE  1 


+ E' 


+ E' 


For  Judge  1,  group  Z and  4 were  combined  into  a common  subspacc 
due  to  the  homogeneity  of  their  mean  criterion  values. 


FIGURE  6-4 


140 


AID  SPLIT  DIAGRAM 
JUDGE  2 


1.  Y=A  U+E<1J 

* v 


2‘  Y=A3XL6(3,4)  ^2X16(1'2)4E(2) 


1 / 2 3 4)  (1) 

3.  Y = A5X10<-  ’ >tA4X,0(  '4A2X16 


4.  Y-A5X,0 


(2,  3.4) 


AI 

48 

N = 224 

A3 

85 

N3  = 108 

A2 

13 

N2  = 116 

1.2)  + e(3) 

> 

U3 

11 

. 91 

N = 91 
b 

A = 

. 53 

N4  = 17 

4 

\ 

4 

,\ 

MA,X 

(I) 

+e{4) 

6 

16 

\ 

A7 

. 38 

N7  = 40 

> 

o 

il 

. 00 

N.  = 76 

6 V 

»•  V = A5Xl012’  '•  4)4  A4X10(,,h  A9X10'4'  3,4A8X1012'  ‘‘W*1 

\ \ A =.68  N„  22  19  \ 


A8  = .l°  Ng  - 21 


6.  Y-A^A^'  l)+A4X10(1)+A9X10(4’  3)+A8X10(2,  1)+A6X16(1)4E(6) 


SS-5  SS-6 


SS-4  SS-3 


‘ 8 TO 
SS?2 


6 16 

"ss^r~ 


Au=.97  Nu  = 68 
A10  r • 74  NIO  = 23 


FIGURE  6-5 


B 


AID  SPLIT  DIAGRAM 
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within  the  subspaces,  first-order  models  tended  to  cross- validate  better. 
The  coefficients  for  the  local  models  were  obtained  by  the  iterative  pro- 
cedure given  in  Chapter  IV.  The  local  models  for  the  subspaces  shown 
in  Figures  6-9  through  6-13,  are  given  in  Tables  6-7  through  6-11. 


SUBSPACES:  JUDGE  1 
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Table  6-7 

Local  Model  Coefficients:  Judge  l 
Constant  = -.6618 

Subspace  1:  "Poor"  Credit  Rating  or  "Other"  Financial  Ref. 

(GM1 B1 ) 

local  model  1;  PS  = .6687  (GM1)  + constant 


Subspace  2:  "No  Bank"  and  not  in  subspace  1 (SSI) 

(GM2=1 ) 

local  model  2:  PS  = .3201  (Credit  Rating)  + . 0722  (Phone) 

+ . 0600  (KesCut.)  + constant 


Subspace  3:  "Good"  Credit  Rating  and  nut  in  SSI  or  SS2 

(GM3  = 1) 

local  model  2:  PS  - 1.  639  (GM3)  l constant 


Subspace  4: 

( GM4  = 1 ) 
local  model  4: 


where: 


Not:  in  SSI,  SS2,  or  SS3 

PS  - . 2258  (Local)  f . 2044  (ResCat)  t . 0901 

(Type  Employment)  1 .0818(CPl)  4 .3312 
(GM4)  t constant 

^.p^  r 1 if  Re  slime  < 2 and  lies  Cat  < 3 
. 0 otherwise 


Best  Cut  Score:  . 58-.  61 

Hit  Rate:  .929 


nwria  irr1 


Table  6-8 


Local  Model  Coefficients:  Judge  2 


Constant  = . 760 


Subspace  1 : 

(GM1  = 1) 
local  model  1 : 


"Poor”  Credit  Rating 

PS  = 7619  (GM1)  + constant 


Subspace  2: 

(GMZ-1 ) 
local  model  2: 


"Unrated"  Credit  Rating  and  No  Checking 
Account 

PS  - 6642  (GM2)  + constant 


Sub space  3: 
(GM3-1) 
local  model  3: 


Sub  space  4: 
(GM4=1) 
local  model  4: 


"Unrated"  Credit  Rating  with  at  least  "Checking" 
Account 

PS  = .0910  (Phone)  -.3271  (Jobtime)  + .0879 
(Loan  Term)  + . 1632  (Financial  Ref.  ) - .0341 
(GM3)  + constant 


Credit  Rating  "Medium"  or  "Good"  and 
"No- Bank"  Account 

PS  = -.  1639  (Type  Employment)  1278  (Income) 
+ . 1602  (Financial  Ref.  ) + constant 


Subspace  5: 

(GM5-1 ) 
local  model  5 : 


Credit  Rating  of  "Good"  or  "Medium",  "Savings' 
or  "Checking"  and  "Major"  Financial  Reference 
PS  = . 211  (GM5)  + constant 


Subspace  6: 
(GM6  = i) 


local  model  o: 


Not  in  any  of  the  other  subspaces 

ro  - . 24ri  (Kesoftt)  t ojU<iu  a elm/ 

-.  3475  Loan  Amt.  -.  0099  (GM6)  + constant 


Best  Cut  Score: 
Hit  Rate: 


.46-  49 
. 929 


SUBSPACES:  JUDCE  3 


FIGURE  6-11 
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Table  6-9 

Local  Model  Coefficients:  Judge  3 
Constant  = - . 201  2 

Subspace  1:  "Poor"  Credit  Rating 

(GM1=1 ) 

local  model  1 : PS  = .2106  (GM1 ) + constant 


Subspace  2:  "Unrated"  Credit  Rating  and  "No  Bank" 

(GM2  = 1 ) 

local  model  2:  PS  = . 2426  (GM2)  + constant 


Subspace  3: 

(GM3=1 ) 
local  model  3: 


"Unrated"  Credit  Rating  and  "Checking"  or 
"Sa  vings" 

PS  = . 0079  (Phone)  + .1099  (Fin.  Ref.  ) 

+ . 2973  (Necessity)  + constant 


Subspace  4: 

(GM4-1 ) 
local  model  4: 


■where: 


"Medium"  or  "Good"  Credit  Rating  and  "Other" 
Financial  References 

PS  = . 1165  (Bank  A.ct.  ) + . 0489  (Credit  Rating) 
+ . 3720  (CPl)  + constant 
f = 1 if  Jobtime  is  > 2 years 
\ = 0 otherwise 


Subspace  5: 

(GM5  = 1 ) 
local  model  5: 


"Medium"  or  "Good"  Credit  Rating  and  "Minor" 

or  "Major"  Financial  References 

PS  = . 2323  (Phone)  + .0571  (Type  Employment) 

+ .1641  (Fir , Ref.  ) + .0180  (Credit  Rating) 
+ constant 


Best  Cut  Score: 
Hit  Rate: 


. 53-. 65 
. 92 
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Table  6-11 


Subspaco  1 : 

(GM1  -1 ) 
local  model  1 : 

where: 


Local  Model  Coefficients:  Judge  5 
Constant  = 5003 

"Poor"  Credit  Rating 

PS  = .5132  (GM1)  -t  . 6865  (CPI)  + constant 

f = 1 Jobtirne  = 5 years  and  "Major"  Fin. 
1=0  Otherwise 


Rif. 


Subspace  2: 
(GM.2---1) 
local  model  2: 


where: 


"Unrated"  Credit  Rating 

PS  = -.  2696  (GM2)  + . 2144  (Marital  ) + . 0452 

(Jobtime)  + .0927  (Fin.  Ref.  ) -.0707  (Phone) 
+ .2936  (CP2)(Phone)  + constant 
r = 1 if  "checking"  or  savings  for  Bank  Acct. 
2 1 = 0 


Subopace  3: 

( GM3  = 1 ) 
local  model  3: 


"Medium"  or  "Good"  Credit  Rating  and  "Minor" 
or  "Other"  Financial  Ref. 

PS  = .1153  (Local)  -.  033(Phone)  +.  l6l6(Restime) 
+ . 0950  (Credit  Rating)(Fin.  Ref.  ) + . 0220 
(ResCat)  + constant 


Sub space  4: 

(GM4=  1 ) 
local  model  4: 


"Medium"  or  "Good"  Credit  Rating,  "Major" 
Fin.  Ref.  and  "Less  than  2 years"  Jobtime 
PS  = . 5221  (Phone)  4 . 1260  (Type  Employment) 
-.013l(GM4)  4 constant 


Subspace  5: 

(GM5  = 1 ) 
local  model  5: 


"Medium"  or  "Good"  Credit  Rating,  "Major' 
Fin.  Ref.  and  "Over  2 years"  Jobtime 
PS  - 1,5011  (GM5)  4 constant 


Best  Cut  Score: 
Hit  Rate: 


.91 

. 44-. 47 


Section  IV: 


Comparison  of  Models: 


For  the  fir  st-order , second-order,  and  local  models,  the 
model  coefficients  shown  in  Tables  6-5  through  6-11  were  used  to  pro- 
duce a prediction  hit/miss  table  of  the  type  shown  in  Exhibits  16  through 
18,  Appendix  B,  This  provided  the  data  to  construct  error  curves 
reflecting  the  number  of  false  approvals  versus  false  denials  for  each 
possible  cut  score  between  . 0 and  1.0.  The  curves  in  Figures  6-14 
through  6-18  reflect  the  profiles  of  these  error  rates  for  each  model 
type  and  each  judge.  The  points  plotted  on  the  curves  reflect  the  error 
rates  at  cut  scores  where  both  the  level  of  false  approvals  and  the  level 
of  false  denial  change.  Thus,  each  point  on  the  curve  is  a local  optimum 
cut  score  since  it  reflects  the  minimum  false  approval  rate  attainable 
at  a particular  false  denial  rate,  or  vice  versa.  The  important  thing 
to  note  from  these  curves  is  that  the  curves  for  the  local  models  domi- 
nate the  curves  for  the  first-order  and  second-order  models;  that  is, 
at  any  given  false  approval  rate,  the  local  model  for  the  judge  will 
result  in  the  same  or  fewer  false  denials  than  his  first-order  or  second- 
order  model. 

In  this  sense  the  "goodness"  of  the  model  is  being  measured 
relative  to  the  model's  predictive  capability  over  the  whole  spectrum 
of  possible  cut  scores  and  not  solely  on  the  model's  maximum  hit  rate. 


Comparison  of  Judges: 


Comparison  of  the  judges'  policies  solely  on  the  basis  of 


FALSE  APPROVALS 
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FALSE  DENIALS 


FIGURE  6-14 


FALSE  APPROVALS 


ERROR  CURVES  FOR  JUDGE  2 


FIGURE  6-15 


FALSE  APPROVALS 
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ERROR  CURVES  FOR  JUDGE  4 
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FIGURE  6-17 


FALSE  APPROVALS 
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ERROR  CURVEa  FOR  JUDGE  5 


FALSE  DENIALS 


FIGURE  6-18 
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the  magnitude  of  the  coefficients  for  the  predictors  is  not  very  informative 
when  the  predictors  arc  correlated,  as  is  the  case  in  this  decision  process. 
Models  with  greatly  different  coefficients  can  result  in  very  nearly  the 
same  set  of  decisions.  For  example,  Table  6-12  presents  the  similari- 
ty f the  decisions  of  the  five  judges  in  terms  of  the  percentage  of  matches. 
Thi.j  data  indicates  that  all  of  the  judges1  decisions  are  the  same  between 
87  and  °1  percent,  yet  Table  6-5  indicates  a fairly  wide  disparity  among 
the  lui  fMcients.  Note  that  for  Judge  2 and  Judge  5,  four  of  the  coeffi- 
cien.  o .i  . illy  have  different  signs,  yel  they  have  thi  highest  match 
on  dei  'si  'is. 


TABLE  6-12 

IN  TE  K JUDGE  HI M 1 LA  H I TV 
(percent) 
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4/5 

Vote 


1 

l 00 

8 0. 

87 

87 

87 

94 

1 

100 
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91 

94. 
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i 

1 00 

88.  5 

80. 
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(X^q).  However,  Bank  Account  is  not  a primary  variable  for  either 
Judge  4 or  Judge  5 while  Loan  Term  (X^)  *s  important  to  Judge  4 

and  Time  on  Job  (X  ) is  important  to  Judge  5. 

5 

Further,  the  Aid-Tree  identifies  a Financial  Reference  of 
"Other"  as  a sufficient  condition  for  denial,  regardles  of  Credit  Rating, 
for  Judge  1.  Yet  Judges  3 and  4 approved  47  percent  and  53  percent  of 
applications  if  they  had  a "Medium"  or  "Good"  Credit  Rating.  Careful 
study  of  tne  Aid- Tree  can  suggest  many  such  differences.  However 
the  question  must  always  be  asked,  "Is  the  difference  due  to  policy  or 
is  it  due  to  the  computational  scheme?  " During  the  research  it  was 
found  that  a very  useful  technique  was  to  present  each  judge  his  tree 
and  hu»c  him  verbalize  or  explain  the  splits.  This  led  to  exposure 
and  discussion  of  many  subtle  differences  in  opinion  among  the  judges. 


6: 


Section  V: 

The  Voting  and  Composite  Models: 

As  was  the  case  in  the  "without  credit"  models,  a voting 
strategy  of  4/5  appeared  to  best  match  the  actual  approvals  and  denials. 

The  last  column  of  Table  6-12  reflects  that  the  decision  of  each  of  the 
individual  j udgc s matches  voting  policy  better  than  they  match  the  decisions 
of  any  of  the  other  individual  judges. 

An  alternate  composite  model  was  also  obtained  by  the  more 
traditional  process  of  clustering  first-order  models  of  the  individual 
judges  with  the  UDSTAT-J  version  of  the  JAN  hierarchiai  clustering 
routine.  The  coefficients  for  the  4/5  vote  and  clustered  composite 
models  are  shown  in  Table  6-13.  figure  6-19  shows  the  merge  diagram 
from  the  clustering  process.  This  diagram  reflects  the  policy  compari- 
son technique  th.it  has  been  generally  used  in  the  past.  It  reflects  that 
Judge  1 and  Judge  ‘>  have  the  must  similar  policies,  yet  their  hit/miss 
ratio  shown  in  Table  0 - 1 2 is  one  ot  the  lowest. 

This  paradox  is  due  to  the  nature  of  the  data  and  the  fact 

that  the  criteria  for  closeness  are  different.  Ln  the  clustering  approach, 

tile  su  u of  squared  errors  between  actual  sc  ire  and  predicted  score  is 

the  criterion.  In  the  hit/miss  approaih,  the  fact  of  a difference  is 

important,  hill  the  magnitudi  is  not  (all  "miss"  magnitudes  arc  1 and 

"hits"  are  0).  Tin-  whole  | Mill  ot  pi  eseiiling  and  ilisiussinj*  the  merge 

diagram  is  that,  with  binary  decisions,  ttie  use  ot  continuous  crit  Tin 

> 

» ui  ll  a s the  s qua  i ei  l multiple  correlation  i net  1 1 ci  cut  (11  ) or  the  .iiini 

ot  squared  errors  is  probably  not  desirable  and  can  lead  to  vastly 
il  it  1 e i enl  r e ■ ail  t s . 
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TABLE  6-13 

I 

COEFFICIENTS  FOR  4/5  VOTE  AND 

COMPOSITE  MODELS 

' 

Predictors: 

Coefficients 

Clustered 

4/5  Vote 

Composite 

Constant 

-1 . 0469 

-. 8615 

Age 

- . 0268 

-.0201 

Ma  r ita  1 

- .0383 

0344 

Local 

. 070  0 

. 0610 

Phono 

. 1393 

. 1053 

J obtimo 

. 0322 

. 0456 

Re  stime 

- . 0308 

-.0279 

Re  sCat 

. 0830 

. 0527 

Type  Employment 

. 0151 

. 01  74 

Rico  me 

- .0225 

0276 

Bank  Account 

. 0655 

. 0584 

Financial  Reference 

. 1538 

. 1279 

Loan  Amount 

. 0072 

.0128 

Loan  Term 

- .0585 

0571 

- 

N e e e s s i ty 

. 006) 

- . O’  46 

Equity 

. 0066 

0093 

Credit  Rating 

. 2647 

. 2622 

Best  Cut  Score: 

. 46 

. 27 

o 

Hit  Hat,  " 

. 94 

- 92 

1 

A, 

. y 

m. 
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' Since  the  4/5  vote  policy  model  actually  represented  a set 

of  substitute  responses  for  the  actual  decisions  or  the  individual  judge's 
decisions,  it  was  used  to  build  an  Aid-Tree  (Exhibit  13,  Appendix  B) 

4 and  to  define  subspaces  and  local  models.  Although  the  suner iority  of 

the  local  model  relative  to  the  first-order  model  still  .>>  Id  up  in  terms  of 
« the  error  curves  shown  in  Figure  6-?.0,  this  supe*-  i j .■  ity  was  not  as 

great  as  it  was  in  the  case  of  the  individual  judges.  This  decrease  is 
■ understandable  since  the  4/5  vote  policy  tends  to  average  the  policies 

,-W* 

''  -T-  and  models  of  the  individual  judge:.,  ai  d iii  •>  ; doing,  obscures  the 

configuralities  and  idiosyncracies  of  Inc  inJ.vidual  policies.  Thus  the 
power  of  the  local  models  in  modeling  tin  arious  subspaces  is  dimin- 
ished since  the  boundaries  of  these  subspaces  art:  not  as  well  defined. 


MERGE  DIAGRAM  FROM 
CLUSTERING  PROGRAM 
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Section  VI. 

Results  of  the  Cross-Validation  Field  Test: 

A Shift  in  the  Sample  Clin rnct eristics  and  the  Pul  icy : 

Ol'  the  1 b 0 new  applications  collected  during  the  field  test, 

75  percent  were  given  loans  without  the  stipulation  of  "full  recourse". 
This  reflects  an  increase  of  ■)()  percent  m the  approval  rate  between 
the  224  app!  ieut  ions  used  in  the  model  building  phased  and  the  150  appli- 
cations of  the  cr oss- validation  population.  This  increase  is  attributable 
to  two  basic  factors  that  are  detectable  in  Table  (j-1  1. 


TAULK  6-1-1 

Al'l'ROVAb  KATi;  vs.  CllKDl'l  UATlffi  (JA I F'iOK  Y 
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* First,  there  was  an  apparent  increase  in  the  caliber  of  the 

applications  relative  to  the  primary  decision  variable,  Credit  Rating. 
Second,  there  was  an  increase  in  the  approval  rate  for  all  categories  of 
Credit  Rating, 

There  are  two  probable  explanations  for  the  increase  in  the 
caliber  of  the  applications: 

1)  sampling  differences  between  the  random  sample  of 
224  and  the  total  July  72  population  of  applications  used 
for  cross-validation 

2)  an  actual  shift  in  the  credit  credentials  of  the  potential 

4 

borrower  during  the  month  of  July  72. 

The  increased  approval  rate  reflects  a change  in  the  rela- 
tionship between  the  dealer  and  the  lending  institution.  This  change 
is  attributable  to  the  longer  time  that  they  had  wurke  1 together  by  the 
time  the  cross-validation  ••ample  was  taken.  Whereas  the  original 
sample  retire  led  their  relationship  alter  only  4 to  b mouths  of  the 
dealer  contract,  they  bail  been  doing,  business  approximately  14  months 
at  the  time  ut  the  second  ■•ample  .nut  the  trust  ml  understanding 
between  them  was  greatly  mipruvid.  During,  this  period,  the  volume 
of  bus  inrun  tr  mi  Dealer  It  bad  grown  by  some  400  peri  ml  and  the  loan 
ol  tic  er  '<  expressed  a willingness  to  accept  sown  wliat  riskier  applications 
in  older  to  kc«  p the  dr. Dei  happy  and  continue  tin  expansion  ut  busi- 
ness. This  i ll.nigc  is  rspri  i.illy  noticeable  in  till  applli  at’oiiii  With 

^ t )i  s I n !i  i on  with  Judge  revealed  tliiic  l‘i  .1  t Uli  I U.lt  loll  ol  the  el  edit 
■ pi.  1 1 i t y lh.it  is  a I ui  1 1 I i < in  ol  the  general  e i o no  m i • condition-..  When 
bit  .Hie:,  s i.  slov  , a general  d«-i  I ease  ot  i redd  ipi.illty  prevails. 
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"Unrated"  and  "Medium"  Credit  Rating. 

"With  Credit"  Policy  Models  vs.  Decisions  on  the  Codified  Predictors: 

One  result  of  these  shifts  was  to  improve  the  performance 
of  the  policy  models  and  lower  the  cut  scores  corresponding  to  the 
optimum  hit  rates.  As  can  be  seen  in  Table  6-15,  the  cross-validation 
hit  rates  for  the  judges'  decisions,  based  on  the  codified  predictor  sets, 
were  generally  be.  or  than  the  hit  rates  on  the  original  sample.  This 


Table  6-15 

MAXIMUM  HIT-RATES 

Equation  Original  Hit-Rate  Cross  Validation  Hit-Rate 


J Utlgc 

1 -- 

first  order 

. 91 

. 96 

J udge 

1-- 

second  order 

. 92 

. 96 

J udge 

1-- 

local  model 

. 93 

. 93 

J udge 

first  order 

. 92 

. 96 

J Ui  1 g i“ 

Z-- 

So  iind  order 

. 92 

. 97 

J udge 

* . 

local  model 

. 93 

. 93 

J udge 

A - - 

first  order 

. 91 

. 95 

J udge 

■j.  .. 

second  order 

. 90 

. 95 

j udgr 

3 

1 > i . a 1 l node  1 

. 92 

. 9 5 

J wdgr 

•l-  - 

first  iniler 

. bH 

. 95 

,1  lldge 

•1  - 

lilt  .\  1 1 1 lot  1 1 1 

. 92 

. 94 

.)  udge 

1 1 1 1*1  Iinlri 

. 90 

. 9<> 

.1  Uil|;i 

lot  .1  1 model 

. 91 

. 9 3 

•l/'i  Vole 

1 1 r ;it  til  • t «■  i 

. 9-1 

. 97 

•i/‘«  \ 

ote 

1 m ,1 1 mode  1 

. 9-1 

. 97 

table  also  reflects  that  the  first-order  and  second-order  models  tended 
to  do  slightly  better  than  the  local  models  in  terms  of  cross-validation. 
Given  the  observed  shift  in  policy,  concentrated  in  the  Credit  Rating 
categories  of  "Unrated"  and  "Medium",  this  is  not  surprising.  These 
are  the  predictor  categories  that  defined  the  subspaces  over  which  the 
local  models  were  the  most  complex  and  influential.  The  superiority  of 
the  local  models  (as  shown  in  Figures  6-14  to  6-19)  was  mainly  due  to 
the  better  fit  of  the  local  hyperplanes  over  these  subspaces.  This 
superior  fit  would  naturally  be  most  affected  by  perturbation  of  the 
policy  in  those  subspaces. 

"With  Credit"  Policy  Mod  el  s vs.  Actual  Decision  s : 

The  various  composite  models  were  cross-validated  against 
the  actual  approvals  and  denials  that  the  loan  officers  had  made  using 
the  original  application  forms  (Exhibit  1,  Appendix  II).  Exhibits  16, 

17,  and  19  present  the  appropriate  hit/miss  tables  and  Figure  6-21 
shows  the  error  curves  for  the  various  policy  models.  The  first-order 
"4/6"  vote  and  composite  models  each  attained  92  percent  accuracy. 

The  local  "4/5"  vote  model  was  91  percent  accurate.  Comparison  of 
the:  first-order  voting  model's  performance  in  predicting  decisions 
that  were  made  on  the  codified  predictors  with  its  performance  in  pre- 
dicting the  actual  decisions  based  on  the  original  application  form  re- 
flects that  there  is  only  a 5 percent  difference.  This  is  somewhat  less 
than  the  difference  between  using  codified  predictors  ami  actual  appLi 
cations  on  the  original  224  as  reflected  by  the  data  from  Table  6-3.  The 
decrease  probably  reflects  the  judges'  ad  ptutinn  to  the  use  ol  single 
category  descriptors  ( hi  ough  then  pa  r t u > pat  in  u in  the  pr  oj e ct . hi  any 
event,  the  ,!  it  t e r envv  between  the  results  based  mi  the  codified  prediitors 
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CROSS-VALIDATION  HIT  RATES  FOR 
VARIOUS  FIRST-ORDER  MODELS 
USING  CREDIT  RATING 


FIGURE  6-21 


172 


and  the  results  based  on  the  natural  language  applications  does  not 
appear  to  be  a significant  problem  in  construction  of  policy  models 
for  this  decision  process. 

C lerical  vs . Expert  Mea suretnents : 

One  of  the  objectives  of  the  field  test  was  to  ascertain  the 
capability  of  the  clerical  staff  to  properly  codify  the  application  data. 

To  this  end.  the  set  of  instructions  in  Exhibit  6,  Appendix  B,  were 
explained  to  the  clerks  and  left  for  their  reference  in  filling  out  the 
applications.  Their  questions  were  solicited  and  answered  the  first 
day  of  the  field  test.  After  each  clerk  expressed  confidence  in  her 
under  standing  of  the  instructions,  there  was  no  further  monitoring  of 
the  process  of  translation  from  the  original  application  into  the  codified 
predictors.  However,  the  judge  who  made  the  decision  on  the  applica- 
tion was  also  asked  to  fill  out  the  coded  application  form.  This  pro- 
vided a means  of  comparing  the  "clerical"  measurement  with  the 
"expert"  mcasiiremom.  An  item-by-item  comparison  on  each  of  the 
applications  was  made  and  the  results  are  displayed  in  Table  6-16. 

The  wide  disparity  in  the  assessment  of  the  predictors  is  rather 
amazing,  especially  in  the  case  of  those  prcdiciu.  s which  should  have 
required  absolutely  no  judgment.  A check  of  both  translations  with 
the  original  applications  revealed  the  loan  officer  was  generally 
correct  in  cases  where  a differenee  existed,  Inn  not  always.  Figure 
6-22  reflects  the  effe*  t of  this  disparity  cm  the'  decisions  predicted  by 
tlu  "4/6"  voting  model.  The  lack  of  a large  effect  in  the  predictions 
is  attributable  tu  the  fact  that  the  primary  predictor  in  the  model  is 
Credit  Rating,  and  this  was  only  available  to,  and  assessed  by,  the 
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Table  6-16 

EXPERT  VS.  CLERICAL  MEASUREMENT  DIFFERENCES 
ON  APPLICATION  VARIABLES 


Predictor 


Cases  Assessed  Differently* 


Age 

10 

Marital  Status 

7 

Local  Family 

25 

Telephone 

12 

J cb  Time 

29 

Residence  Tim  ■ 

29 

Residence  Category 

26 

Type  Employment 

61 

Income 

52 

Bank  Account 

40 

Financial  References 

69 

Loan  Amount 

32 

Loan  Term 

16 

Necessity 

18 

Equity 

35 

Credit  Rating 

Only  assessed  l 

*At  least  one  item  differed  on  141  of  the  ISO  t ppl  icaticr.s . 


loan  officer,  in  the  case  of  the  ’’tentative  approval 
Rating  was  not  used.  A comparison  of  Figures  6-23  and  6-24  reflects 
that  misassessing  the  predictors  does  have  a substantial  effect.  In 
either  case,  it  would  appear  that  some  formal  training,  of  the  clerical 
staff  would  be  required  to  insure  that  their  assessment  of  the  predictors 
is  eongvuen*  with  that  of  the  loan  officers. 


"Without  Credit"  Policy  Models  vs,  Actual  Decisions: 

The  shift  toward  leniencv  in  the  policy  of  the  judges  had  a 
positive  elteet  on  the  observed  performance  oi  the  policy  models  used 
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CROSS-VALIDATION  HIT  RATES  VS. 
MEASUREMENT  TYPE 


FIGURE  6-22 
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to  predict  "tentative  approval".  Figure  6-23  reflects  that  if  everyone 
scoring  . 85  or  above  had  been  approved,  over  48  percent  of  the  applica- 
tions could  have  been  "tentatively  approved"  with  only  a 3 percent  error 
rate.  This  is  well  above  the  25-30  percent  criterion  set  by  the  loan 
officers  as  a minimum  limit  for  usability.  Further,  in  assessing  the 
mistakes  that  occurred,  it  was  found  that  2 of  the  3 false  approvals, 
made  by  the  model  at  the  . 85  cut  score,  had  actually  been  granted  under 
the  "full  recourse"  option.  Figure  6-24  indicates  the  performance  of 
the  model  with  clerically  translated  predictors  is  significantly  worse 
and  would  probably  be  unacceptable.  Figure  6-25  shows  that  the  model 
which  includes  "Dings"  is  somewhat  more  conservative,  but  does  per- 
form well  in  terms  of  error  rate.  As  for  implementation,  the  use  of 
the  more  conservative  model  would  probably  be  the  better  option  in 
order  to  assure  acceptable  performance  when  the  overall  credit  quality 
is  lower. 

First-Order  vs.  Local  Models : 

The  reversal  of  the  first-order  and  local  models  in  terms 
of  cross-validation  hit  rate  would  indicate  that  the  local  model  is  more 
sensitive  to  policy  changes,  especially  if  the  change  only  affects 
certain  subgroups  of  the  applicants.  However,  the  local  model  is  more 
robust  in  the  sense  that  it  has  a better  profile  of  hit  rates  over  the 
entire  range  of  application  scores,  and  would  be  less  sensitive  to 
general  changes  in  the  policy  affecting  the  entire  spectrum  of  applica- 
tions. Figure  6-26  presents  the  profile  of  hit  rates  on  the  cross- 
validation  sample  for  all  possible  cut  scores.  The  first-order  models 
enjoy  a local  superiority  at  a given  cut  score;  the  local  models  do  a 
better  "average"  job  of  predicting  the  decisions. 
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RISK  CURVES  FOR  “4/5"  POLICY 
FOR  TENTATIVE  APPROVAL 
(EXPERT  MEASUREMENT) 
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Section  VII. 

Implications  for  Implementation: 

The  results  of  this  modeling  effort  indicates  that  there  are 
several  applications  of  the  Policy  Capturing  technique  that  would  be 
beneficial  to  the  installment  loan  decision  process. 

Training: 

The  discussion  and  interaction  that  the  process  stimulated 
among  the  loan  officers  was  felt  to  be  very  beneficial.  In  a post  project 
interview,  there  was  general  agreement  among  the  loan  officers  that  the 
models  helped  them  to  understand  how  their  colleagues  thought  and  to 
illuminate  some  of  the  more  subtle  aspects  of  their  own  policies.  One 
specific  example  of  this  improved  communication  was  the  definition  and 
clarification  of  the  concept  of  Financial  Reference.  A second  example 
was  best  expressed  by  one  of  the  junior  loan  officers,  who  said: 

I really  did  not  know  how  the  three  senior  loan 
officers  evaluated  the  applications,  other  than 
on  past  credit,  until  they  discussed  their  'trees'. 

It  would  appear  that  having  a new  employee  make  decisions  on  a 
standard  set  of  loan  applications  and  then  capturing  his  policy  to 
compare  with  those  the  existing  group  of  loan  officers  would  be  a 
simple  and  effective  method  of  evaluating  his  compatibility.  An 
example  of  this  was  encountered  when  the  transferrability  effort  was 
pursued.  The  policy  of  a new  employee  with  considerable  past  experi- 
ence as  manager  of  another  loan  company  was  captured. 
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The  results  reflected  that  he  would  have  given  some  38  percent 
of  the  applicants  with  "Poor"  Credit  Rating  loans,  while  the  corresponding 
success  rate  for  the  "4/5"  vote  was  0 percent.  Further,  his  understanding 
of  the  newly  defined  predictor  of  Financial  Reference  and  his  concern  with 
Bank  Account  were  not  commensurate  with  the  attitudes  of  the  other  loan 
officers.  The  policy  capturing  exercise  made  early  identification  of 
these  differences  possible. 

Mechanization  of  Prescoring  or  "Surrogate"  Judgment : 

The  performance  of  the  "with  credit"  models  appears  to  be 
sufficient  to  allow  their  use  either  as  an  aid  to  the  loan  officer,  or  to 
automatically  approve  loans.  Although  Dawes  and  Diller's  numerical 
criterion  for  indicating  "bootstrapping"  will  occur  does  not  appear  to  be 
directly  applicable  in  the  case  of  binary  decisions,  the  comparison  of  the 
hit  rates  of  the  models  (See  Table  6-15.  ) with  the  reliability  of  the  judges 
(Sec  Table  6-3.  ) would  indicate  that  any  of  the  models  would  provide 
some  aid. 

Procedures  for  implementing  the  models  could  be  manual 
or  computerized.  Any  of  the  equations  previously  presented  could  be 
accessed  through  a computer  terminal  or  stored  on  a programmable 
desk  calculator  to  provide  the  application  scor  Figures  6-27  and  6-28 
present  the  risk  curves  tlmt  could  be  used  by  the  loan  officer  as  he 
considered  an  application.  On  those  applications  that  do  not  contain 
significant  extenuating  circumstances,  a fixed  cut  score  could  be  used 
for  automatic  approval  or  denial.  Figure  6-28  indicates  that  by 
evaluating  applications  with  the  local  "4/5"  voting  model,  using  a rule 
of  granting  loans  which  scored  over  . 06  and  denying  loans  that  scored 
less  than  . 04  would  represent  minimum  risk  and  yet  leave  a very  small 
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RISK  CURVES  FOR  ‘4/5"  POLICY 
(LOCAL  MODEL)  WITH  CREDIT  RATING 


FIGURE  6-28 
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range  of  ambiquity. 

One  ancillary  advantage  of  vising  the  policy  models  is  that 
the  loan  officer  could  get  a consensus  evaluation  from  his  colleagues  by 
using  their  models  along  with  his  own. 

" T entative  Ap proval'1  During  Non-banking  Hours : 

The  performance  of  the  "tentative  approval"  models  indicates 
that  a definite  improvement  in  the  level  of  service  the  lender  can  provide 
to  the  dealer  could  accrue.  The  motivation  for  use  of  these  models  lies 
in  the  speed  advantages  that  the  use  of  the  models  provides.  Figure 
6-29  shows  the  distribution  of  times  that  the  cross-validation  applica- 
tions spent  in  the  "system"  and  the  corresponding  times  they  spent  in 
the  "credit  system".  The  difference  in  the  average  times,  14.  3 hours, 
is  largely  due  to  the  lack  of  access  the  dealer  has  to  the  lender  during 
non-banking  hours.  The  tentative  approval  of  applications  could  have 
eliminated  this  waiting  time  for  48  percent  of  those  "aftor-banking-hours" 
applications  submitted  during  the  field  test.  This  is  particularly  appli- 
cable to  sales  made  on  Friday  afternoon  or  Saturday,  but  which  must 
wait  until  Monday  for  consideration. 

Again,  cither  manual  or  computerized  techniques  could  be 
used.  In  the  simplest  system,  a manual  of  predictor  categories  with 
application  scores  (similar  to  a telephone  book)  could  bo  provided  to 
the  dealer.  This  approach  would  be  rathei  inflexible  to  policy  changes. 

A more  sophisticated  system  could  utilize  a programmable  desk  cal- 
culator or  a remote  computer  terminal  for  real-time  evaluation  of 
the  application. 

The  difference  between  the  models  for  "surrogate"  judg- 
ment and  "tentative  approval"  lies  in  the  current  non-availability  of  a 
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credit  rating  at  the  dealers  shouromii.  With  current  trends  in  automation, 
this  condition  v.ill  likely  change  with  the  credit  history  of  an  applicant 
being  stored  in  the  credit  bureau  auii|nit-T  and  being  accessible  electroni- 
cally via  a remote  computer  terminal.  In  this  situation,  much  better 
evaluations  could  be  made  and  the  connotation  of  ''tentati  ve  approval" 
could  be  removed.  To  implement  a system  of  this  type,  the  policy  model 
of  the  lending  institution  could  be  stored  in  a time-sharing  computer. 

At  the  time  the  application  is  made,  it  could  be  tvped  into  a remote 
terminal  such  as  a teletype,  the  credit  rating  could  be  electronically 
obtained  from  the  credit  bureau,  and  the  application  score  and  an 
appropriate  decision  could  be  rendered.  An  ancillary  benefit  would 
be  the  electronic  transmission  of  the  original  application  to  all  appro- 
priate parties  and  the  elimination  of  much  of  the  manual  effort 
currently  required  from  the  clerical  staffs  of  the  lender  and  the  dealer. 

Transferrability  of  Policy  Models: 

Two  experienced  loan  off'cers  from  other  lending  institu- 
tions wore  used  to  assess  the  potential  transferability  of  the  predictor 
set  and  the  models  ih  veloped  for  the  loan  officers  at  lanuir  Savings. 

In  initial  interviews,  both  of  these  judges  reviewed  the  predictors 
and  their  categorizations,  and  generally  agreed  with  them.  Although 
each  expressed  some  differences  in  the  exact  categorizations  of 
various  predictors,  these  diffi  rences  were  no  greater  than  the 
differences  that  had  existed  among  the  three  senior  loan  offieers 
from  Ixxmar  Savings  at  the  beginning  cr  the  project.  The  most  impor- 
tant thing  noted  was  that  the  only  variables  suggested  by  these  judges 
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that  were  not  included  in  the  16  predictors  shown  in  Exhibit  3,  Appendix  B, 
were  Sex  and  Number  of  Dependents.  Both  of  these  were  suggested  by 
Judge  7.  He  noted  that  his  clientele  was  composed  mainly  of  salaried 
employees  with  large  families  from  minority  groups  and  was  somewhat 
more  restricted  and  less  affluent  than  the  clientele  of  lamar  Savings. 
Neither  of  the  predictors  were  viewed  by  Judge  7 as  critical  omissions. 

The  AID- t rees  for  the  two  judges  are  shown  in  Exhibits  14 
and  15  of  Appendix  B.  A comparison  of  their  decisions  with  the  vote 
decisions  of  Judges  1 through  5 is  given  in  Table  6-17. 


TABLE  6-17 

COMPARISON  OF  "4/5"  VOTE  AND  DECISIONS  OF 
JUDGES  6 and  7 


J ud  g c 

No /No 

Yes/No 

No/Yes 

Yes/Yes 

Percent  Match 

6 

61 

4 

66 

93 

69 

7 

98 

8 

29 

89 

83.  5 

Both  the  AID- Tree  and  the  hit  table  reflect  that  Judge  6 
apparently  lias  a different  policy  than  that  reflected  by  the  majority 
of  the  original  loan  officers.  The  lower  predictive  capability  of  the 
AID- Tree  would  suggest  that  the  policy  of  Judge  6 may  not  be  as  well 
defined  or  consistent  as  those  of  the  other  judges.  The  data  in  the 
hit  table  reflects  a greatly  different  policy  toward  denials.  As  previ- 
ously discussed,  Judge  6 had  just  been  hired  by  Lamar  Savings  from 
his  position  as  manager  of  a smaller,  more  speculative  loan  company. 
The  high  number  (66)  of  approvals  by  Judge  6 which  were  denials  by 
the  4/5  criterion  would  indicate  that  Judge  6 was 
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accustomed  to  assuming  more  risk  by  granting  applicants  with  lower 
Credit  Rating  loans.  His  results  represent  a fortuitous  demonstration 
of  the  potential  use  of  the  Policy  Capturing  technique  to  uncover  dif- 
ferences in  philosophy  and  background  of  new  employees. 

The  AID- Tree  and  hit  table  for  Judge  7 reflect  that  his 
policy  is  much  closer  to  the  "4/5"  vote  of  the  original  judges,  but  there 
are  still  some  differences,  primarily  in  the  importance  of  Bank  Account 
and  Financial  References,  These  differences  are  probably  reflective 
of  the  different  clientele  since  Judge  7's  policy  is  better  explained  by 
variables  such  as  Residential  Category,  Time  at  Residence,  and  Income, 
than  it  is  by  variables  such  as  Financial  References  and  Bank  Account. 

It  is  important  to  note  that  the  predictive  capability  of  the  AID-Tree  for 
Judge  7 is  comparable  with  those  of  the  other  five  judges.  This  indicates 
that  the  omission  of  Sex  and  Number  of  Dependents  as  predictors  does 
^ not  greatly  degrade  the  model  and  confirms  his  assertion  that  they  are 

not  critical  to  his  policy. 

Based  on  this  rather  limited  insight  into  transferability, 
the  practice  of  using  the  weights  of  a policy  model,  derived  for  one  en- 
vironment, in  another  environment  does  not  appear  to  be  the  best 
approach.  However,  the  redefinition  of  these  weights  by  using  a 
standard  set  of  predictor  eases  does  appear  feasible. 
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Section  VIII 
Summary: 


This  chapter  has  presented  the  major  results  of  the  effort 
to  model  the  policy  of  installment  loan  officers  under  two  conditions-- 
with  and  without  access  to  a Credit  Rating  on  the  applicant.  Results  of 
both  local  and  first-order  models  suggest  that  they  could  be  effectively 
employed  to  aid  the  loan  officers  in  making  more  efficient  and  timely 
decisions.  The  use  of  the  AID- Tree  was  found  to  be  a particularly 
good  catalyst  to  interjudge  communication,  particularly  when  the  judges 
are  not  accustomed  to  conceptualizing  decision  processes  in  terms  of 
algebraic  or  statistical  models. 

The  results  suggest  that  the  local  models  are  more  idiosyn- 
cratic than  the  first-order  models  and  are  sensitive  to  policy  changes 
that  affect  only  certain  portions  of  the  decision  space.  However,  this 
sensitivity  could  be  reduced  by  using  a larger  set  of  data  to  define  the 
local  models.  The  local  models  do  provide  a better  "average"  fit 
over  the  entire  space. 

Models  that  were  able  to  predict  binary  decisions,  using 
codified  predictors,  with  97  percent  accuracy  were  developed.  In 
predicting  decisions  made  with  the  natural  language  applications,  92 
percent  accuracy  was  attained.  In  either  case,  these  levels  of  pre- 
dictability arc  better  than  the  90  percent  reliability  of  the  judges. 

When  compared  to  the  75  percent  accuracy  that  would  result  if  every 
applicant  during  July  72  had  been  given  a loan,  the  improvement  in 
accuracy  obtainable  with  the  policy  models  ranges  between  17  and  22 
percent.  Whereas  giving  everyone  who  applies  a loan  is  clearly  unwise 
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in  the  viewpoint  of  the  loan  officer,  this  latter  comparison  is  not  really 
too  informative.  The  actual  economic  advantage  of  using  the  policy 
models  can  only  be  evaluated  after  they  are  used  in  a decision  making 
role  and  appropriate  cost  and  profit  data  has  been  collected.  Comparison 
of  such  with  the  costs  and  profits  for  the  current  mode  of  operation  will 
then  reflect  the  economic  viability  of  using  the  models. 

The  feasibility  of  identifying  the  top  30  percent  of  the  appli- 
cations in  the  absence  of  the  primary  predictor  was  demonstrated. 

Finally,  evidence  of  the  utility  of  the  Policy  Capturing 
technique  to  assess  interjudge  differences  and  evaluate  the  policy  of 
new  employees  was  obtained. 


CHAPTER  VII:  CONCLUSIONS,  COMPARISONS, 

AND  RECOMMENDATIONS 

Viability  of  Applying  Policy  Capturing  to  Operational  Decisions: 

The  primary  goal  of  this  research  was  the  adaptation  and 
demonstration  of  Policy  Capturing  in  operational  environments.  Toward 
this  end,  the  analysis  was  performed  entirely  with  data  taken  from  an 
operational  environment  and  the  criteria  for  success  were  defined 
relative  to  the  needs  and  conditions  within  that  environment. 

The  ultimate  product  sought  was  a technique  that  could 
be  used  by  the  Operations  Research  practitioner  to  define,  evaluate, 
compare,  and  predict  the  judgments  of  decision  makers  in  the  business, 
academic,  and  administrative  fields.  In  order  for  a modeling  technique 
to  be  viable  in  these  environments,  it  must  be  reasonably  straightforward 
to  use  and  compatible  with  the  stimulus  data  (in  amount  and  format)  as 
it  exists  in  the  environment  being  modeled.  It  must  produce  models 
that  are  accurate  and  understandable. 

The  results  embodied  in  Table  6-15,  Figure  6-23,  and 
Figure  6-28,  and  the  remarks  of  the  junior  loan  officer,  (page  180  ) 
are  probably  the  best  evidence  that  can  be  offered  for  the  utility  and 
viability  of  Policy  Capturing  in  the  operationally  valid  environment 
that  was  studied  in  this  research.  They  reflect  that  policy  models 
were  obtained  which  could  predict  the  decisions  of  installment  loan 
officers  with  92  percent  accuracy  when  all  of  the  important  predictors 
are  available.  Models  were  also  produced  which  could  meet  the 
criterion  of  identifying  the  top  25-30  percent  of  the  loan  applications 
even  when  the  most  important  predictor  was  not  available.  The 


remarks  of  the  junior  loan  officer,  and  the  continued  participation 
of  the  already  burdened  staff  of  the  installment  credit  department  attest 
to  the  belie vability  and  communicability  of  the  model. 

Comparison  of  a judge's  policy  as  expressed  by  any  of  the 
algebraic  models  presented  in  Chapter  VI,  with  the  "structural  image" 
model  as  expressed  by  the  corresponding  AID-Tree,  leaves  little 
doubt  as  to  the  benefits  of  graphically  displaying  the  policy.  This  is 
especially  important  in  communicating  the  concept  of  Policy  Capturing 
to  decision  makers  who  are  not  mathematically  inclined. 

Modeling  Configural  Judgment  Processes  with  Local  Models: 

As  for  the  conceptualization  and  modeling  of  configural 
decision  processes  as  a series  of  local  hyperplanes,  the  comparison 
of  the  models  obtained  for  the  loan  officers  (see  Figures  6-14  through 
6-18)  would  appear  to  support  the  superiority  of  this  approach  from 
both  the  accuracy  and  the  interpr etability  standpoints.  Collateral 
support  for  the  contention  that  interaction  in  judgment  processes  might 
best  be  modeled  as  a discrete  phenomen?  is  provided  by  the  comparison 
of  the  results  obtained  using  local  model  with  Valenzi' s xesults  for 
"continuous"  models. 

The  Character  of  the  Model  Seeking  Technique,  AID: 

In  assessing  the  robustness  of  the  model  seeking  procedure 
embodied  in  the  application  of  the  AID4UT/AIDTRE  programs,  several 
pertinent  points  can  be  made.  First,  categorical  variables  are  used; 
this  is  the  most  general  of  all  possible  representations  of  the  individual 


predictors.  Second,  there  are  nc  underlying  restrictions  or  assumptions 
imposed  in  the  model  seeking  phase  of  the  process.  Third,  the  procedure 
does  not  require  designed  experiments  or  independent  predictors.  Fi- 
nally, the  capacity  of  the  computer  program  is  large  enough  to  handle  up 
to  80  possible  predictors  and  does  not  require  an  "a  priori"  culling  of 
the  variables,  or  hypothesis  of  the  interactions. 

On  the  other  hand,  the  procedure  does  not  produce  a 
completely  definitive  answer,  but  rather,  requires  the  application  of 
considerable  judgment  on  the  part  of  the  analyst  in  the  interpretation 
of  the  output  data.  In  this  sense,  the  technique  merely  provides  the 
analyst  with  the  capability  of  choosing  where  he  wants  to  inject  his  wisdom 
and  experience  in  the  modeling  process.  With  more  traditional  analysis 
techniques,  he  makes  the  decision  by  choosing  the  technique  and  impli- 
citly imposing  the  appropriate  assumptions.  In  the  use  of  AID4UT/ 
AIDTRE  and  local  models,  he  makes  his  decisions  in  terms  of  explicit 
interpretations  of  which  subspaces  and  local  models  to  use. 

Comparison  of  Results  of  this  Research  with  Those  of  Past  Research: 

There  are  some  interesting  comparisons  that  can  be  made 
between  the  results  of  this  research  and  the  conclusions  of  past  investi- 
gators. These  comparisons  generally  reflect  that  the  policy  models 
developed  for  the  loan  officers  are  in  harmony  with  the  results  and 
expectations  of  the  clincial  psychologists. 

In  reference  to  past  attempts  to  define  configural  models, 
the  resuits  of  this  research  reflect  an  improvement  that  is  chiefly 
attributable  tc  the  use  of  the  AID  technique  for  determining  where  to 
"fish"  (Green,  1968)  for  configural  relationships  and  the  modeling  of 
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these  interactions  as  discrete  phenomena. 

Other  specific  comparisons  include: 

Power  of  First-Order  Models,  vis-a-vis,  Configural  Models.  Hoffman 
(1960)  and  Yntema  and  Torgerson  (1961)  demonstrated  the  power  of 
first-order  paramorphics  in  modeling  judgment  processes.  The  results 
of  this  study  reflect  *he  same  situation.  Purely  on  the  basis  of  statis- 
tical significance,  there  is  probably  no  difference  between  the  performance 
level  of  the  various  types  of  models  investigated  in  this  research.  However 
given  the  choice  of  two  equally  good  models  (in  terms  of  either  explained 
variation  or  hit  rate),  the  model  that  appears  to  be  more  reflective  of 
the  underlying  process  would  be  the  better  choice  since  it  would  be  more 
intuitively  satisfying  to  the  judge. 

Relative  Shrinkage  of  Configural  Models  Due  to  "Overfit".  Goldberg 
(1968)  found  that  configural  models  of  judgment  processes  tend  to  be 
far  less  stable  than  their  corresponding  first-order  models.  Ward 
(1954)  indicated  that  configural  models  generally  tended  to  "overfit"  the 
idiosyncracies  of  the  data,  as  indicated  by  much  greater  shrinkage  in 
the  cross-validation  coefficient.  From  these  findings,  one  could  infer 
that  if  the  configural  model  held  up  as  well  as  the  fir3t-order  model 
upon  cross-validation,  some  evidence  of  legitimate  configural  effects 
exists.  The  shrinkage  for  the  first-order  models  of  the  loan  officers 
averaged  8.  7 percent  while  the  shrinkage  for  their  local  models 
averaged  9 7 percent.  This  difference  of  only  1 percent  would  indicate 
that  the  "overfit"  phenomena  is  minimal.  It  would  offer  evidence  that 
the  configurality  is  real,  especially  since  the  cross-validation  r;stes 
themselves  were  uniformly  higher  for  the  local  models  (see  Table  *a-4). 
Diminished  Configurality  in  Group  Policies.  Slovic  (196R1  noted  that 
models  for  group  judgments  tended  to  be  less  configural  than  those  lor 
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individuals.  This  same  situation  is  reflectei  by  comparison  of  the 
differences  in  predicted  capability  between  the  first-order  models  and 
local  models  for  the  "4/5"  vote  policy  with  the  corresponding  differences 
for  the  individual  loan  officers.  (See  Table  6-15.) 

AID- Tree  Representations  of  Einhorn's  Non-compensatory  Models . The 
lexicographic  model  suggested  by  Einhorn  (1970)  is  conceptually 
similar  to  the  process  depicted  by  the  structure  of  an  AID- Tree.  His 
conjunctive  model,  in  which  strong  attributes  are  most  important  in 
determining  the  total  score,  is  analogous  to  the  substitute  advantage 
model  cf  Sonquist  (1964)  which  results  in  an  upper  terminating  trunk- 
twig  structure  in  AID-Trees.  Similarly  his  disjunctive  model  correlates 
to  the  substitute  disadvantage  model  as  reflected  in  the  lower  terminating 
trunk-twig  structure. 

Recommendations  for  Further  Research: 

This  research  has  served  to  demonstrate  the  potential 
utility  of  applying  Policy  Capturing  to  a practical  decision  process, 
but  it  does  not  represent  a final  answer  to  the  applicability  of  the 
concept.  The  project  has  probably  generated  more  ideas  and  questions 
than  it  lias  resolved.  These  new  questions  can  provide  the  basis  for 
further  research  projects  in  both  the  areas  of  1)  development  of  new 
applications,  and  Z)  improvement  of  the  current  modeling  techniques 
and  capabilities.  Specific  projects  recommended  by  this  investigator 
are: 

Implementation  of  Policy  Models  a « Decision  Aids  for  Loan  Officers : 

The  models  presented  in  this  dissertation  could  be  implemented  on 


either  a programmable  desk  calculator,  or  a remote  teletype  hookup  to  a 
time- sharing  computer.  The  primary  tasks  of  such  a project  would 
include: 

!)  Development  of  the  computer  code  or  calculator  routine 
to  implement  the  appropriate  models  and  evaluate 
incoming  applications. 

2)  Redesign  of  the  loan  application  form  to  facilitate 
collection  of  the  appropriate  categorical  data. 

3}  Tiaining  of  the  clerical  staff  for  proper  evaluation  and 
coding  of  the  predictor  data. 

4)  Collection  and  evaluation  of  data  to  evaluate  actual 
performance  of  the  policy  models  in  terms  of  the 
"ex-post"  quality  of  the  approvals  and  the  lost  revenue 
of  the  denials. 

5)  Expansion  and  modification  of  the  current  models  to 
allow  implementation  for  other  appliance  dealers  and 
the  development  of  models  for  other  categories  of 
dealer  contracts. 

Sequential  Generation  of  Stimulus  Data:  At  the  beginning  of  this  research, 

the  incremental  generation  and  consideration  of  single  cases  of  stimulus 
data  was  envisioned  as  a possible  method  of  constructing  policy  models 
with  a minimal  set  of  data.  At  this  point  it  is  still  unclear  how  signifi- 
cant changes  and  trends  in  the  policy  could  be  detected  by  addition  of  a 
single  new  decision  at  each  stage  of  the  Policy  Capturing  process. 
However,  a modified  version  of  this  concept  of  incremental  data 
generation  does  appear  to  have  merit  when  used  with  the  AID4UT/AIDTRE 
computer  programs  in  the  definition  of  subspaces  and  local  models. 
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In  development  of  a policy  model,  initial  analysis  should 
be  performed  with  a sample  of  actual  stimulus  data  of  at  least  10 
cases  per  predictor.  The  identification  of  homogeneous  subgroups 
would  eliminate  sub^paces  within  the  predictor  space  which  do  not 
require  further  investigation.  In  th'  non-homogeneous  groups,  past 
experience  indicates  that  there  are  often  too  few  data  units  to  determine 
a stable  model.  This  problem  could  be  solved  by  generation  of  an  incre- 
mental set  of  synthetic  stimulus  data  which  falls  entirely  within  the  non- 
homogeneous  subspace.  In  this  manner,  the  data  set  would  be  augmented 
in  those  subspaces  of  the  predictor  set  where  the  policy  of  the  judge  is 
most  complicated  and  in  which  there  is  a requirement  for  more  pre- 
dictors and  more  complicated  local  models. 

The  task  of  incrementally  generating  realistic  stimulus 
data  within  selected  subspaces  could  be  approached  by  using  a synthetic 
data  generation  routine  such  as  the  one  developed  by  R.  F.  Fallis  of 
the  Ohio  State  University  Psychology  Department.  This  routine  was 
used  by  Naylor  and  Wherry  (1964)  to  demonstrate  the  feasibility  of 
capturing  policies  with  synthetic  data.  The  correlation  matrix  obtained 
from  the  original  set  of  actual  decision  data  would  be  used  by  the  data 
generator  to  assure  realistic  data  was  generated  within  the  subspaces 
under  consideration. 

A specific  research  project  in  this  area  would  include  the 
following  tasks: 

1)  Acquisition  and  modification,  or  development, 
of  a synthetic  data  generation  routine. 


* 


'.,4T 


% 


2)  Identification  of  an  appropriate  decision  process  and 
application  of  the  proposed  procedure  for  defining 
local  models.  For  example,  synthetic  data  for  the 
non-homogeneous  subspaces  of  the  loan  officer's 
AID- Trees  might  be  generated  and  new  models  genera- 
ted. 

3)  Evaluation  of  the  change  in  the  model  quality  resulting 
from  the  synthetic  augmentation  procedure. 


Estimation  of  lime  to  Repair  Electronic  Equipment:  During  the  past 

decade,  the  process  of  estimating  the  mean  time  to  repair  (MTTR)  of 
various  electronic  equipment  has  become  of  considerable  concern  to 
design,  reliability,  and  maintainability  engineers.  The  need  for  this 
data  is  growing  increasingly  important  for  use  in  optimization  models 
for  modularizing  equipments  and  determining  repair  strategies.  The 
problem  lies  in  the  fact  that  the  MTTR  data  is  usually  needed  in  the 
early  stages  of  equipment  development,  but  is  not  available  until 
after  the  equipment  has  been  produced  and  become  operational.  Only 
after  the  equipment  has  failed  and  been  repaired  can  observational 
data  be  gathered  and  analyzed.  Even  if  this  process  were  efficient, 
which  it  is  not,  the  resultant  data  is  obsolete  by  the  time  it  is  available. 
The  past  procedure  for  obtaining  MTTR  data  needed  in  the  design 
process  has  been  to  infer  values  for  new  equipments,  based  on  experi- 
ence with  similar  equipments.  This  process  has  not  been  as  accurate 
as  is  required  for  use  in  sophisticated  optimization  and  design  routines. 
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A lecent  study  was  sponsored  by  the  U.  S.  Air  Force 
Materials  Laboratory,  WPAFB,  Ohio,  in  which  the  subjective  estimates 
of  MTTR  values  for  23  various  pieces  of  electronic  equipments  were 
obtained  from  experienced  maintenance  technicians.  Although  this 
study  had  some  of  the  aspects  of  a Policy  Capturing  effort,  it  did  not 
model  the  technicians  judgments  as  a function  of  the  basic  characteristics 
of  the  equipment.  Instead,  estimates  for  each  individual  piece  of  equip- 
ment were  collected  and  compared  with  actual  maintenance  experience 
for  that  piece  of  equipment.  The  results  indicated  ■?,  . 86  cori-elation 
between  the  estimates  of  the  repair  times  and  the  actual  repair  times 
(Smith,  1971),  This  reflects  that  technicians  can  make  valid  estimates 
of  repair  times  on  particular  equipments,  but  it  does  not  improve  the 
situation  relative  to  predicting  the  repair  times  for  new  and  different 
equipments. 

It  would  appear  that  this  situation  offers  a natural  area  for 
the  application  of  policy  models.  A project  in  which  the  estimated 
repair  times  for  each  equipment  were  modeled  as  a function  of  the  basic 
characteristics  of  the  equipment  instead  of  specific  equipment's  names 
or  stock  numbers  might  provide  the  model  that  could  be  used  for  pre- 
dicting MTTR  values  for  new  equipments.  The  project  would  consist 
of  modeling  the  MTTR  data  in  terms  of  parameters  which  were  common 
among  equipments  and  building  models  with  these  parameters  as  pre- 
dictors. Specific  tasks  to  be  performed  in  such  a project  would 
include: 

1)  Determination  of  the  basic  characteristics  of  a 

class  of  equipments  such  as  function,  size,  shape, 
type  of  assembly,  accessibility,  type  of  failure 
mode,  repair  procedure,  and  required  test  equipment. 
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2)  Acquisition  of  existing  judgmental  data  from  the  ap- 
propriate agency  and  description  of  the  individual 
equipments  in  terms  of  these  basic  characteristics, 

or  generation  of  new  judgment  data  on  new  equipments. 

3)  Definition  of  models  for  estimating  MTTR  values  and 
comparison  of  the  predicted  values  for  similar  equip- 
ments, not  U3ed  in  the  modeling  phase,  with  actual 
maintenance  data  that  is  available  in  such  publications 
as  the  FARADA  Handbooks. 

Prediction  of  Successful  Rehabilitation  of  Military  Prisoners:  A project 
that  could  be  of  direct  benefit  in  further  evaluating  the  potential  of  the 
local  modeling  technique  proposed  in  this  dissertation  is  suggested  by 
a study  performed  by  Smith,  Gott,  and  Bottenberg  (1967).  In  that 
study,  various  trait  and  background  data  was  collected  on  U.  S.  Air 
Force  prisoners  and  first-order  models  for  predicting  successful 
return  to  duty  were  developed.  They  achieved  predictive  efficiencies 
of  77.  4 percent  and  suggested  that  better  models  could  probably  be 
obtained  with  further  analysis.  Their  primary  objective  was  the 
demonstration  of  feasibility  of  the  Policy  Capturing  approach. 

During  the  course  of  the  current  research,  the  need  for  such 
a predictive  system  surfaced  as  a result  of  discussions  with  the  person- 
nel from  the  U.  S.  Army  Correctional  Training  Facility  (CTF),  Fort 
Riley,  Kansas.  In  their  case,  accurate  prediction  of  potential  failures 
and  successes  of  retrainees  could  be  used  to  modify  the  length  of  stay 
and  the  course  of  instruction  required  for  individual  retrainees.  Such 
predictions  would  conceivably  be  made  on  the  basis  of  instructor 
evaluations  obtained  in  the  early  part  of  the  rehabilitation  program. 

This  project  could  also  provide  insight  into  the  proposed 
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use  of  Policy  Capturing  techniques  to  evaluate  differences  in  policy 
between  similar  governmental  agencies.  The  experience  of  interacting 
with  two  separate  governmental  agencies  would  undoubtedly  draw  upon 
the  managerial  talents  of  the  researcher  as  well  as  his  technical  talents 
and  would  be  a particularly  appropriate  project  for  an  operation 
researcher  or  management  scientist. 

Specific  tasks  of  this  project  would  include: 

1)  Acquisition  of  the  appropriate  data  used  by  Smith, 

Gott,  and  Bottenberg.  This  data  is  still  on  file  at 
the  Personnel  Research  Laboratory,  AFHR.L, 
Lackland,  AFB,  Texas. 

2)  Analysis  of  data  with  the  AID4UT/AIDTRE  programs 
and  the  development  of  local  models. 

3)  Comparison  of  the  local  models  with  these  first-order 
models  obtained  in  the  previous  study  by  Smith,  Gott, 
and  Bottenberg  (1967). 

4)  Acquisition  and  modeling  of  judgment  and  actual  data 
from  the  CTF,  Fort  Riley,  Kansas. 

5)  Comparison  of  the  data  and  models  on  an  inter-agency 
basis. 


Final  Comments! 

The  ultimate  goal  of  this  research  effort  was  the  "demon- 
stration of  the  potential  benefits  and  the  viability  of  applying  the  Policy 
Capturing  concept  in  ecologically  valid  environments".  To  be  totally 
complete,  such  a demonstration  must  include  more  examples  than  were 
reasonably  accomplishable  during  the  16  months  spent  on  this  research. 
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Fortunately,  the  examples  attempted  were  successful,  knowledge  was 
gained,  and  tools  developed  which  should  make  further  demonstrations 
of  the  concept's  viability  less  laborious. 

The  projects  suggested  for  further  research  lepresent  areas 
in  which  this  investigator  believes  significant  results  might  be  produced 
in  a relatively  short  time.  Other  applications  are  somewhat  more 
esoteric  and  will  require  considerably  more  conceptual  innovation. 
Examples  of  further  applications  of  Policy  Capturing  are  the  development 
of  models  for  the  objectives  and  desires  of  voters,  preference  models 
and  consumer  policies  in  marketing,  and  the  modeling  of  the  objective 
functions  for  research  and  development  planners. 

The  results  of  this  experiment  have  increased  the  interest 
in  Policy  Capturing  and  the  research  will  undoubtedly  continue.  However, 
there  is  much  to  be  accomplished  before  Policy  Capturing  becomes  a 
completely  "canned"  technique. 

Whatever  the  final  outcome  of  these  proposed  projects, 
some  benefit  will  inevitably  be  realized  for  these  applications  of 
Policy  Capturing.  As  a minimum,  the  process  of  trying  to  quantify 
subjective  human  judgment  forces  the  participating  decision  maker  to 
think  about,  and  verbalize,  his  policy.  In  this  respect,  the  perception 
that  benefits  could  be  derived  from  mathematically  modeling  decision 
processes  is  not  new.  As  early  as  1889,  Lord  Kelvin  recognized  the 
potential  of  quantifying  human  judgment  by  stating: 

I often  say  that  when  you  can  measure  what  you  are  speaking 
about,  and  express  it  in  numbers,  you  know  something  about 
it,  but  when  you  cannot  measure  it,  when  you  cannot  express 
it  in  numbers,  your  knowledge  is  of  a meager  and  unsatisfac- 
tory kind;  it  may  be  the  beginning  of  knowledge,  but  you  have 
scarcely,  in  your  thoughts,  advanced  to  the  stage  of  science, 
whatever  the  nature  may  be. 


* 

APPENDIX  A 

Definition  and  Comparison  of  Mathematical  Models 


Purpose: 


In  the  literature  covering  the  past  research  in  Judgment 
Analysis,  various  mathematical  models  are  discussed  but  they  are 
seldom  defined  in  explicit  mathematical  terms.  One  of  the  goals  of 
the  present  research  is  the  introduction  of  the  model  seeking  technique, 
AID,  as  a tool  for  improving  the  policy-model  hypothesis  process  and 
transforming  that  process  into  more  of  a directed  search  and  somewhat 
less  of  a "fishing  expedition".  Since  the  many  subtle  aspects  of  linear 
modeling  theory  cannot  all  be  discussed  in  this  appendix,  the  objective 
of  this  appendix  is  to  merely  discuss  the  various  models  and  computa- 
tional methodologies  considered  in  this  and  past  research  and  to  relate 
them  to  the  general  linear  model. 

The  basic  function  of  any  model  is  the  parsimonious 
description  of  nature  Models  can  be  classified  as  being  either 
deterministic  or  stochastic.  In  Policy  Capturing  we  are  concerned  with 
stochastic  models  and  the  use  of  various  statistical  techniques  to  build 
and  evaluate  stochastic  models.  The  model  itself  represents  the  rela- 
tionships that  are  presumed  to  exist  in  the  population.  Any  assumptions 
that  are  mar>r  as  adjuncts  to  the  model  pertain  to  the  population. 

The  General  Linear  Model:  (GLM)1 


*The  reader's  familiarity  with  vector  and  matrix  notation  and  termi 
nology  will  be  assumed  throughout  this  discussion. 


« 
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A general  linear  model  is  a model  in  which 


wher  e: 


X0  + e or  Expected  Value  (Y)  = Y = X g 


an  'n  x 1'  vector  of  random  observa- 
tions from  the  population  of  all  possible 
Y values 


xnxiz“xi 

Y • • • • 

X21 

• • • • t 

X * * * * 

nl 


an  'n  x p*  matrix  of  fixed 
quantities 


a ' p x 1'  vector  of  unknown 
parameters 


an  'n  x 1'  vector  of  random 
errors 


The  GLM  for  a Hypothetical  Three- Predictor  Problem: 


One  can  always  formulate  a problem  as  a general  linear 
model  which  has  unique  binary  predictors  for  each  possible  combina- 
tion of  the  predictor  variables.  For  example  consider  a problem  in 
which  there  is  a criterion  that  is  a function  of  three  predictor  variables, 

Y + f(R.,  S.,  T ),  two  of  which  (R,  S)  are  ordinal,  and  the  third  (T) 
ijk  l j k 

being  categorical.  Further,  consider  that  there  are  three  distinct 

levels  associated  with  each  of  these  variable  and  that  the  data  to  be 

2 

analyzed  consists  of  one  point  having  each  of  the  possible  combinations 
of  these  predictor  values.  Table  B-i  shows  the  criterion  value  for 
each  of  the  27  possible  combinations  (patterns)  of  the  predictor  variables. 


TABLE  B- 1 


f- “ 

i=l 

R 

i=2 

R 

i=3 

H 

T 

k=2 

T 

k=3 



T 

k=l 

T 

k=2 

T 
k = 2 

T 

k=3 

s 

i=i 

vm 

Ynz 

Y 1 1 3 

Y 21 1 

Y21 2 

Y213 

Y311 

Y 31  2 

Y 3 1 3 

s 

Y 1 21 

Y 1 22 

Y1Z3 

Y221 

Y222 

Y223 

Y321 

Y322 

Y323 

S 

Y13I 

Y132 

Y 1 3 3 

y 

231 

Y232 

Y233 

Y 3 3 l 

Y332 

Y 3 3 3 

The  most  general  linear  model  that  could  be  written  for  pre- 
dicting the  value  of  a data  units  possessing  one  of  fhese  27  predictor  com- 
binations would  be  of  the  form: 


2 

In  the  general  case,  multiple  data  points  would  occur  for  each  pre- 
dictor-combination. In  this  case,  each  criterion  value  would  have 
an  additional  subscript  denoting  that  it  was  the  one  of  several  in  the 
(i,  j,  k)'th  cell. 
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Y = p X +S  X + 
ijk  1 111  P2  112 


&26  X332  + S27  X333  + 6ijk 


(A) 


where:  Y... 


the  criterion  value  for  that  data  point  having  the 
i'th  level  of  predictor  R,  the  j'th  level  of  predictor 
S,  and  the  k'th  level  of  predictor  T.  (the  criterion 
value  for  that  point  in  the  (i,j,k)'th  cell) 

r 1 if  the  point  is  in  the  (i,  j,  k)'th  cell 
1 0 otherwise 


^ (n=l,27)  = the  27  unknown  parameters  of  the  model 

n 


the  error  in  the  predicted  value  of  the  point  in  tne 
(i,  j,  k)1  th  cell. 


In  vector  notation  corresponding  to  that  given  in  the  definition  of  the 
GLM,  Y would  be  a vector  of  27  criterion  values,  X would  be  a 27  by 
27  identity  matrix,  and  B would  be  a vector  of  27  unknown  parameters. 

Note  that  the  model  that  we  have  presented  is  not  very 
interesting  in  that  it  has  exactly  the  same  number  of  parameters  as 
there  are  data  patterns.  Further,  there  is  only  one  point  per  cell  and 
each  data  point  can  be  predicted  exactly:  therefore,  the  error  vector 
is  null.  This  model  is  not  particularly  desirable  ir,  terms  of  the 
parsimony  with  which  it  describes  the  data  but  it  does  serve  as  an 
example  of  a true  model;  a true  model  being  defined  as: 


A model  that  does  not  impose  any  relationships  upon 
the  expected  values  of  the  criterion  variable  which  do 
not  exist  in  nature. 


Although  a model  which  has  one  binary  predictor  per 
variable  pattern  is  a sufficient  condition  for  a model  to  be  "true", 
it  is  not  a necessary  condition.  A more  parsimonious  "true" 
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model  might  be  attainable  if  certain  regularities  exist  in  the  relationships 
between  the  individual  predictors  and  the  criterion  values.  Such  relation- 
ships represent  restrictions  on  the  GLM  that  can  be  explicitly  imposed 
on  the  model,  but  which  are  more  commonly  implicitly  "assumed11  to 
exist  in  the  process  of  model  formulation. 

Equivalency  and  tho  Regression  Formulation  of  the  GLM: 

Eefore  preceding  with  the  discussion  of  the  restrictions 
that  are  imposed  on  models  as  an  alternative  to  representing  each 
possible  predictor  pattern  by  its  own  binary  vector,  the  concept  of 
equivalent  models  will  be  introduced.  Two  linear  models  (Ml  and 
M2)  are  said  to  be  equivalent  if  they  meet  the  following  conditions: 

1)  Both  models  have  the  same  criterion  vector:  (Y) 

2)  Both  models  have  the  same  number  of  linearly  inde- 
pendent predictors:  (rank  of  X.  , = rank  of  X.,_) 

M l M2 

3)  Every  vector  in  the  first  model  is  a linear  combina- 
tion of  the  vectors  in  the  second  mod  el. 

The  multiple  regression  model  is,  in  fact,  equivalent  to 

the  general  linear  model.  In  the  regression  formulation,  the  X-matrix 

is  composed  of  an  n-dimensional  unit  vector  and  a 'n  x (p-  1 ) ’ matrix 

of  pr»  dictor  values.  The  most  general  formulation  of  the  regression 

model  is  tho  "categorical"  or  "binary  regression"  model.  The  binary 

regression  model  results  when  the  elements  of  the  X-matrix  only  takes 

on  the  values  of  0 or  1,  and  the  predictor  vectors  are  interpreted  as 

classifications  or  attributes.  In  this  model,  the  x values  of  0 or  1 

np 

reflect  the  absence  or  presence  of  thep'th  attribute  in  the  n'th  data 
case. 


The  binary  regression  model  that  is  eqni valent  to  equation 
(A)  i'or  the  hypothetical  problem  is: 

Y...  = Bn  + eiXu  .+  B>X,  + + e-u  (B) 

nk,  0 1 111  2 1 12  26  332  nk 


where: 


Y = as  defined  previously 
ijk 

X. = as  defined  previously 
ijk 


= 027'  P1  " ei"0O'  e2  " ^2  + 0O  “ 


. etc. 


In  model  (B),  the  GLM  of  model(A)has  been  reparameterized  and  the 

new  parameters  have  been  referenced  to  the  value  of  Y^  as  a result 

of  the  elimination  of  the  0 coefficient.  There  are  still  27  parameters 

with  which  to  estimate  27  data  points,  thus  allowing  the  prediction  of 

the  Y,„  values  without  error, 
ijk 

The  regression  model  given  in  (B)  could  be  written  in  terms 
of  the  vector  notation  given  in  the  definition  of  the  GL.M.  In  that  case, 

Y would  be  the  same  vector  of  27  criterion  values,  X would  be  a 
partitioned  matrix  of  the  form 


X = [ u;A.] 


where: 


U = ai  27  by  1 unit  vector 
I = a 26  by  26  identity  matrix 
O - a 1 by  26  null  row  vector 
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Because  of  the  generality  of  the  multiple  regression  model 
and  its  equivalence  to  the  general  linear  hypothesis,  all  other  linear 
models  or  computational  techniques  can  be  viewed  as  special  cases  of 
the  regression  model  upon  which  various  restrictions  have  been  im- 
posed. Each  of  these  special  computational  techniques  can  be  accom- 
plished by  applying  multiple  regression  procedures  to  equivalent 
regression  models  that  have  been  obtained  by  reparameterization  of 
the  general  linear  model.  Although  the  mathematical  equivalence  of 
various  experimental  designs  ard  special  computational  models  to 
reparameterized  regression  models  has  long  been  recognized,  the 
specific  techniques  and  procedures  to  effect  such  reparameterizations 
have  only  recently  been  addressed  by  such  authors  as  Bottenberg  and 
Ward  (1963),  Draper  and  Smith  (1966),  Jennings  (1967),  Mendenhall 
(1968),  Weber  (1971),  and  Ward  and  Jennings  (in  press). 

Restrictions  Imposed  on  the  GLM: 

As  mentioned  previously,  there  are  several  types  of 
restrictions  that  are  commonly  placed  on  the  general  version  of  the 
regression  formulation  of  the  GL.M.  Searle  (1971)  has  pointed  out 
that  any  restriction  can  be  viewed  from  three  different  viewpoints. 
These  three  viewpoints  are: 

1)  The  restriction  is  arbitrarily  imposed  as  a side- 
condition  solely  for  the  purpose  of  obtaining  a unique 
solution  to  the  parameters. 

2)  The  restriction  is  naturally  generated  as  a result 
of  the  hypothesis  being  tested  and  the  presumed 
state-ot'-nature. 
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3)  The  restriction  is  part  of  the  basic  model. 

In  this  sense,  restrictions  placed  on  the  GLM  to  achieve 
parsimony  would  fall  into  category  2.  The  two  most  common  of  these 
restrictions  are: 


'■'■■■'A 

f. 


■X 


Constant  Difference  Restrictions:  This  restriction  implies  that  the 

a 

difference  in  the  expected  value  of  the  criterion  value  (Y)  is  a constant 
(K)  for  any  unit  difference  in  the  value  of  the  predictor  variable.  In 
terms  of  our  hypothetical  model  this  would  mean  that  if  there  were  a 
unit  difference  between  the  1st,  2nd,  and  3rd  levels  of  the  variable  R, 
the  following  relationships  would  hold: 


1 


= K 


2 


A A <k  A 

Y -Y  * Y -Y 

121  221  221  321 


9 


A A A A 

Y -Y  = Y -Y 

133  233  233  333 


This  restriction  allows  the  representation  of  all  levels  of  the  R variable 
by  a single  predictor  and  thus  reduces  the  number  of  predictor  variables 
in  the  model.  By  assuming  both  of  the  ordinal  variables  (R&S)  conform 
to  the  constant  difference  restriction,  the  model  for  the  hypothetical 
problem  could  be  written: 


0n+0.R +e,S.  + B_  T.  + ST.,  + B.R.S. 

0 1 i 2 j 3 k=l  4 k=2  5 i j 

+ 0,  R.  T,  ...  +0_R.T  +0  R.T  +0  S.T. 

o i K-i  ( i k=2  8 i k=3  9 j k=l 

+3  ST  +3  S.T,  +9,  R.S.T,  t 
p10  j k=2  11  j k^S  p12  i j k=l 

+3., R.S.T  +3  R.S.T  + e . (C) 

13  i j k=2  14  i j k-3  ljk 


where: 


R.  = the  actual  value  ojt  R at  the  i'th  level 
1 

S.  = the  actual  value  of  S at  the  j'th  level 
J 

_ f 1 for  data  units  at  the  1st  level  of  T 
k~l  1 0 otherwise 

_ f 1 for  data  units  at  the  2nd  level  of  T 
k=2  1 0 otherwise 


T 


k=3 


r 1 for  data  units  at  the  3rd  level  of  T 
l 0 otherwise 


If  the  assumption  were  true,  then  equation  (C)  would  still  represent  a 
"true"  model. 

No-interaction  (Additivity)  Among  the  Predictors:  This  restriction 

implies  that  the  difference  in  the  expected  value  of  the  criterion  value 
as  a result  of  a difference  in  the  value  of  one  predictor  is  independent 
of  the  value  of  the  other  predictors;  that  is: 

« A A A A «•  » 

Y -Y  = Y -Y  = Y -Y  . =Y  -Y 

111  211  121  221  131  231  133  233 

etc. 

If  this  type  restriction  is  imposed  on  the  model  for  the  hypothetical 
example,  the  resulting  model  would  be: 


ijk  0 1 i=l  2 i-2  3 i-3  4 j-1  5 j-2  6 j-3 


+e7Tk=l+e8Tk=2+  Cijk 


(D) 


1 for  data  units  at  the  first  level 

where:  R.  . , S.  , , T = { of  the  predictor 

i=l  i = l k=l  L ..  . 

0 otherwise 


213 
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R..v 

1=2 


S.__, 

J-2 


k=2 


1 for  data  units  at  the  2nd  level  of 
the  predictor 
0 otherwise 


1 for  data  units  at  the  3rd  level  of 
the  predictor 
0 otherwise 


If  both  the  constant  difference  and  additivity  restrictions 
are  imposed  upon  the  GLM,  the  model  takes  on  its  most  parsimonious 
form: 


y...  = + + t +b.t  + « 

ijk  0 1 i 2 j 3 k-1  4 k=2 


ijk 


where: 
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R. 


S. 
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k=2 


actual  value  of  R at  the  i'th  level 
actual  value  of  S at  the  j'th  level 

r 1 for  data  units  at  the  1 st  level  of  T 
\ 0 otherwise 

r 1 for  data  units  at  the  2nd  level  of  T 
I 0 otherwise 


" * 


.Ox 


M' 

» af 

, d ■ 1 

" • I 

M ; 

" XS  1 


The  Estimation  of  Model  Parameters  and  Evaluation  of  Model  "Truth" 
from  Sample  Data: 

The  hypothetical  example  that  was  previously  given 
considered  a population  that  had  only  27  data  points,  and  a "true 
model"  could  be  written  very  easily.  Generally,  models  are  used  to 
describe  the  relationships  in  much  larger  populations  of  data  with 


many  more  predictor  patterns.  A parsimonious  mode]  that  is  also  known 
to  be  true  may  not  be  readily  apparent.  In  this  case,  the  analyst  must 
rely  upon  probability  and  statistical  theory  to  evaluate  the  hypothesis 
that  any  particular  model  is  “true".  This  involves  the  evaluation  of 
the  parameters  of  the  model  itself  as  well  as  the  statistics  associated 
with  the  particular  analytical  technique  that  is  being  employed. 

In  order  to  estimate  the  parameters  of  a model,  sample 
data  is  taken  from  the  population  and  analyzed  with  the  appropriate 
statistical  techniques.  The  model  itself  specifies  the  structural  re- 
lationships among  the  variables.  The  sample  data  is  used  to  infer 
the  values  for  the  parameters  of  the  model.  The  quality  of  these 
inferences  is  a function  of  the  deviation  of  the  hypothesized  model  from 
a "true  model"  and  the  degree  to  which  the  sample  data  resembles  the 
population.  Neither  the  degree  of  "truth"  of  a model  nor  the  amount 
of  sampling  error  can  be  assessed  exactly.  However,  use  of  appro- 
priate analytical  and  experimental  procedures  can  obviously  help  to 
avoid  the  imposition  of  "untrue"  relationships  and  can  help  to  minimize 
sampling  error.  In  the  first  case,  avoidance  of  restrictions  and 
assumptions  which  impose  arbitrary  relationships  on  the  variables  will 
reduce  the  chance  of  imposing  erroneous  relationships.  In  the  second 
case,  the  use  of  randomization  and  the  selection  of  samples  that  are 
sufficiently  large  will  help  to  reduce  sampling  error, 

3 

In  order  to  distinguish  between  the  parameters  for  the  population 
and  the  estimates  of  these  parameters  derived  from  the  sample,  the 
convention  of  using  Greek  letters  to  designate  population  parameters 
and  the  corresponding  English  letters  to  designate  the  sample  esti- 
mates will  be  followed. 
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Given  a model  and  sample,  the  analyst  generally  has  a 
choice  of  which  statistical  analysis  technique  he  wishes  to  use.  This 
choice  is  tempered  by  the  amount  and  type  of  data  and  the  particular 
information  that  the  analyst  wants. 

The  specific  statistical  analysis  techniques  to  be  discussed 
in  this  appendix  include:  Multiple  Regression  (MR),  Analysis  of 
Variance  (ANOVA),  Multiple  Classification  Analysis  (MCA),  and 
Automatic  Interaction  Detection  (AID'.  All  of  these  formulations  are 
actually  restricted  cases  of  the  general  linear  model,  and,  in  that 
sense,  they  might  best  be  viewed  simply  as  alternative  analytical 
techniques,  each  of  which  is  applied  to  data  fitting  a particular  experi- 
mental design  and  each  of  which  most  conveniently  achieves  some 
analytical  goal. 

The  most  important  difference  among  these  analytical 
techniques  is  not  in  the  basic  character  of  the  underlying  models; 
rather,  it  is  in  the  way  in  which  those  models  are  obtained.  In  this 
context,  the  differentiation  of  the  techniques  into  "model  seeking" 
and  "model  hypothesis"  categories  is  appropriate.  MR,  ANOVA,  and 
MCA  fall  into  the  category  of  "model  hypothesis"  and  AID  falls  into 
the  category  of  "model  seeking". 

A "model  hypothesis"  technique  is  one  in  which  a model  of 
the  structural  relationships  among  the  individual  variables  of  the 
problem  is  hypothe sired  on  an  a priori  basis  and  the  analysis  is 
performed  to  determine  the  relevance  of  the  individual  variables. 

A "model  seeking"  teennique  is  one  in  which  the  model  is 
not  hypothesized  on  an  a priori- basis,  but  rather,  the  data  is  analyzed 
and  displayed  in  a form  so  as  to  allow  model  hypothesis  to  proceed. 
Model  seeking  techniques  are  generally  heuristic  and  mathematically 


216 


informal,  such  as  in  the  case  of  trial  and  error  approaches.  The  AID 
technique  represents  a step  toward  formalization  of  the  model  seeking 
process  based  on  the  analysis  of  the  data  with  a sequential  variance- 
reducing  procedure. 


* 


The  Least-Squares  Computational  Process: 

The  estimation  of  the  parameters  for  a multiple  regression 

model  has  generally  been  confined  to  the  procedures  of  least-squares 

n 2 

in  which  the  sum  of  the  squared  residual  errors  (ESS  = £ e.  ) is 

i=l  1 

minimized.  This  procedure  involves  the  explicit  or  implicit  solution 

of  a set  of  ' p'  simultaneous  equations,  known  as  the  normal  equations, 

4 

for  'p'  parameters  (b.),  that  are  the  estimates  of  the  model  parameters 

(fU. 

Standard  computational  formulae  for  the  parameters  are 
derived  and  given  in  many  texts  such  as  Draper  and  Smith  (1966). 

They  are: 

Given:  Y = XM  c 

the  least- squares  estimator  of  P is: 

b = (X'X)-1  X'Y 


1 


4 

Here  ' p*  refers  to  the  number  of  parameters  and  includes  the  constant 
'b0'  and  the  'b;'  weight  of  each  of  the  predictor  v.  riables  of  the  prob- 
lem. 


t 


217 


the  predicted  values  of  the  criterion  are: 


Xb 


the  residuals  are: 


e = Y - Y 

and  the  coefficient  of  determination  is: 


R 


2 


b'X'Y  - ((  E y.)2/n) 
i=l  x 


Y'Y  - <(  2 y.)2/n) 
i=l  1 


and  the  Error  Sums  of  Squares  is: 


ESS  = Y’Y  -b'X'Y' 


There  are  several  algorithms  for  solving  these  normal 
equations,  each  of  which  has  some  particular  advantages.  The  most 
direct  of  these  algorithm  involves  explicit  solution  of  the  above  equations 
as  given  in  their  matrix  form.  This  procedure  has  one  major  drawback, 
if  the  X'X  matrix  is  singular,  there  exists  no  inverse  and  the  equations 
cannot  be  solved.  This  situation  exists  where  there  are  linear  depen- 
dencies in  the  set  of  predictor  vectors  and  the  number  of  parameters 
in  the  model  (p)  is  greater  than  the  number  of  unique  patterns  in  the 
data  set  (m).  In  cases  where  p > m,  the  X-matrix  is  said  to  be  of 
less  than  full  rank  and  a unique  solution  for  the  parameters  estimates 


is  unobtainable  . A less  direct  approach  embodies  the  iterative  deter- 
mination of  the  parameters  and  does  not  require  matrix  inversion.  If 
the  predictors  are  independent,  this  procedure  will  result  in  the  same 
parameter  values  as  the  explicit  solution,  and  if  the  predictors  do 
contain  dependencies,  it  will  produce  one  of  the  infinitude  of  non-unique 
solutions  that  corresponds  to  the  minimum  ESS.  Thus,  use  of  this 
latter  procedure  will  allow  acquisition  of  the  ESS  information  required 
for  hypothesis  testing  relative  to  full  and  restricted  models  even  when 
the  individual  parameters  cannot  be  uniquely  determined. 

A computer  code  that  accomplishes  the  first  procedure  is 
the  STEP  01  routine  of  The  University  of  Texas  Center  for  Highway 
Research,  and  a code  that  accomplishes  the  second  procedure  is  the 
REGREJ  routine  of  the  EDSTAT-J  Library  (Jennings,  1971)  at  the  UT 
Computation  Center. 

A Numerical  Example  for  Comparing  the  Various  Analytical  Techniques: 

In  the  comparison  of  the  various  analytical  techniques,  a 
small  numerical  example  is  now  presented.  The  data  for  this  example 
is  given  in  Table  B-Z.  The  general  linear  model  corresponding  to 
equation  (A)  is  given  in  Table  B-3.  The  corresponding  data  for  the 
binary  regression  model  is  given  in  Table  B-4.  Table  B-5  reflects 
the  data  for  the  model  in  which  the  predictors  (X^.X^.X^)  have  been 
restricted  to  by  the  "constant  difference"  relationship  discussed 
previously.  It  does  not  reflect  imposition  of  the  "additivity"  restriction. 
Note  that  the  very  small  change  in  the  error  sum  of  squares  (ESS) 
between  the  models  in  Table  B-4  and  Table  F 5 would  indicate  that  the 
resti  iction  does  not  substantially  detract  from  the  "truth"  of  the  model. 
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TABLE  B-2 

DATA  FOR  NUMERICAL  EXAMPLE 


_ V 

x, 

= 2 

V‘ 

— 

rsl 

tt 

OJ 

X 

V 

X?  = 2 

V1 

12.  5 

20.  5 

3 

9 

X=3 

36.  5 

-1 

9 

II 

r^i 

X 

30.  5 

44.  5 

-3 

9 

TABLE  B-3 

GENERAL  LINEAR  MODEL  FOR  NUMERICAL  EXAMPLE 


$ 


YT  = (12.  5,  24.  5.  30.  5.  20.  5,  35.  5,  44.  5,  3,  -1,  -3,  9,  9.  9) 


1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

l) 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1 

bT  = 0T  a (12.  5.  24.  5,  30.  5,  20.  5,  36.  5,  44.  5,  3,-1,  -3,9,  9,  9) 


= (0,0,  0,0,0,  0,0,  0,0,  0,0,0) 


ESS  = 0 

Note  that  p=12(number  of  predictor s)/n=l 2 (number  of  data  points) 


m 
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TABLE  B-4 

BINARY  MULTIPLE  REGRESSION  MODEL 
FOR  NUMERICAL  EXAMPLE 


Y = (12.  5,  24.  5,  30.  5.  20.  5,6.  5.  44.  5,  3,  -1.  -3,  9,  9,  9) 


X 


11  oooooooooo 
101000000000 
1001  00000000 
100010000000 
100001000000 
100000100000 
10000001  0000 
100000001000 
100000000100 
100000000010 
1 00000000001 
1 00000000000 


bx  = eT  = (9,  3.  5,  15.  5,  21.  5,  11. 5,  27.  5,  35.  5,  - 6,  -10,  -12,  0,0) 

«T  = (0.0.  0.0.  0,0,  0,0,  0,0,  0,0) 

ESS  = 0 


f 


TABLE  B-5 

REGRESSION  MODEL  WITH  CONSTANT  DIFFEREN  CE 
PREDICTORS  FOR  NUMERICAL  EXAMPLE 
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®0U+B!X1  +e2X2  + 93X34S4XlX2+e5Xl  W2X3  + e) 
(12.  5,  24.  5,  30.  5.  20.  5,  36.  5,  44.  5,  3,  -1,  -3,  9,  9,  9) 


0 


X 


1111111 

1113  13  3 

1114  14  4 

112  12  12 
1 1 2 3 2 3 6 

1 1 2 4 2 4 8 

12  112  2 1 
1 2 1 3 2 6 3 

1 2 1 4 2 8 4 
1 2 2 1 4 2 2 
1 2 2 3 4 6 6 
1 2 2 4 4 8 8 


bT  = (1. 24.  .038,  7.  19,  11. 79, -1.70, -7.  99,  2.  12) 
T 

e = e2' e3' c4' €5' e6’ e7’ *8’ €9*  C10’ el  !' *i2) 


ESS(Error  Sum  of  Squares)*  = . 1433 


T 

* The  data  for  the  b and  ESS  were  calculated  with  EDSTAT-J 
computer  program- -see  Table  B-8.  The  ESS  is  explained  on  the 
following  pages. 


H. 


The  Analysis  of  Variance  Technique: 


The  Analysis  of  Variance  (ANOVA)  technique  is  a computa- 
tional methodology  which  directly  apportions  the  total  variance  within 
a set  of  experimental  data  into  that  part  of  the  variance  that  is  attribu- 
table to  differences  in  the  levels  of  the  predictor  variables  (treatments) 
and  that  part  that  is  due  to  random  error  within  the  treatment  levels. 
ANOVA  is  a model  hypothesis  technique  that  is  usually  directed  at  the 
testing  of  the  null  hypothesis  that  the  mean  responses  for  each  of  the 
treatment  levels  GO  are  equal,  i.  e.  : 

H:  = *2  = ^3  = •"  = *p 

There  is  no  single  model  formulation  for  ANOVA;  the  model 
tested  in  any  given  analysis  is  a function  of  the  >.nderlying  experimental 
design  of  the  data  and  the  hypothesis  being  tested.  Since  a discussion 
of  the  many  aspects  of  experimental  design  is  not  within  the  scope  of 
this  appendix,  the  discussion  henceforth  will  be  confined  to  the  simplest 
of  all  ANOVA  formulations.  The  case  to  be  discussed  is  the  fixed- 
effects  model  for  a single  factor  experiment  in  a completely  random- 
ized design.  Excellent  references  on  both  Experimental  Design  and 
ANOVA  are  Kempthorne  < i 952),  Hicks  (1964),  and  Mendenhall  (1968). 

The  One-Way  ANOVA  Formulation : 

Since  the  objective  of  analysis  of  variance  procedures 
is  to  test  the  hypothesis  of  the  equality  of  means,  the  linear  model 
for  the  analysis  i3  formulated  in  terms  of  deviations  of  the  mean 
response  for  each  of  the  treatments  from  the  overall  mean  response 


or  grand  mean.  These  differences  are  known  as  "effects".  The  model 
for  the  experiment  under  discussion  is: 


where:  Y.. 

»J 


Y. , ^ u + a.  + «..  (E) 

ij  ) ij 


: response  of  the  i'th  observation  at  the  j'th 
treatment  level 


: grand  mean 


LA  . 

i? 


V 

E 


i=l  j = l 


or. 

) 


effect  of  treatment  "j" 


: random  error  of  the  i'th  observation  at  the  j'th 
treatment  level 


If  the  numerical  example  problem  v/ere  to  be  formulated  as 
a one-way  ANOVA  problem  on  the  variable  X^  (by  ignoring  the  and 
dimensions  of  the  predictor  set)  there  would  be  6 data  points  at 
each  level  of  X^.  The  appropriate  vectors  for  the  model  are  shown  in 
Table  B-6. 


TABLE  B-6 

NUMERICAL  PROBLEM  AS  ONE-WAY  ANOVA 


YT  = {12.  5,  24.  5,  30.  5,  20.  5,  36.  5,  44.  5,  3.  -1,-3,  9,  9,  9) 


T 

e 


X 


T 


1 l 1 1 1 1 1 1 1 1 1 1 

111111000000 
000000111111 


<C11*  C21’  *31’  *41'  *51’  *61*  *12’  C22' €32’ ®42’ ®52' e62) 
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Note  that  there  are  only  two  unique  data  patterns  with  which 
to  estimate  the  three  parameters,  u,  a and  a . Hence  m<  p and  the 

X b 

situation  exists  in  which  a unique  value  for  the  minimum  ESS  can  be 
found  but  unique  values  of  the  parameters  cannot  be  found.  Whereas 
the  specific  purpose  of  ANOVA  is  generally  the  comparison  of  and 
a , this  is  an  unacceptable  situation.  In  actuality,  the  model  that  was 

fa 

presented  in  equationlE)  is  not  complete.  The  additional  restriction  of 


P 

I 


a. 

J 


0 


(F) 


is  also  imposed  on  the  model.  This  restriction  is  listed  by  many 
authors  as  an  assumption  and  its  actual  function  is  not  realized  by 
many  students.  In  reference  to  Searle's  (1971)  categorization  of 
restrictions,  both  viewpoints  2 and  3 appear  to  be  proper  interpre- 
tations and  the  ANOVA  model  is  actually  composed  of  both  equations 
E and  F. 

Since  the  ANOVA  technique  is  directed  at  testing  of  hypo- 
thesis and  ultimately  results  in  the  application  of  statistical  sigriifi- 
cance  tests,  the  statistical  assumption  that  = NID(0,  la  ) is 
generally  imposed. 

Considerations  in  ANOVA  for  Multiple  Predictor  Experimental  Designs : 

In  the  application  of  ANOVA,  certain  simplifications  in 
interpretation  accrue  when  the  total  variance  in  the  data  set  can  be 
divided  directly  into  independent  portions.  This  can  be  accomplished 
with  simplified  mathematics  if  the  experimental  design  for  the  data 
set  is  factorial  with  equal  cell  frequencies.  Although  the  absence 


4r' 

» 

' 


p. 

' /' 

y-“-‘ . 

'*  V 

'Jte 
. »** 

>.••-  • k 

.»;  •*  ' 


of  this  computation -\1  nicety  does  not  prohibit  application  of  AlNOVA,  it 
does  diminish  the  advantage  of  testing  the  hypothesis  of  equality  of  means 
via  ANOVA.  In  the  latter  case,  the  test  of  hypothesis  by  comparisons 
of  the  ESS  obtained  from  full  and  restricted  regression  models  is  a much 
more  attractive  alternative.  Again  it  is  to  be  emphasized  that  the 
relationship  between  an  ANOVA  formulation  and  a specific  regression 
formulation  of  any  problem  is  dependent  upon  the  hypothesis  to  be  tested 
and  the  experimental  design  for  the  problem 

As  an  example  of  the  equivalence  of  the  data  obtained  by 
the  traditional  calculational  procedures  for  ANOVA  and  the  use  of  full 
and  restricted  regression  models,  consider  the  data  for  our  numerical 
problem.  Tabic  B-7  shows  the  ANOVA  table  that  resulted  from  hand 
calculations  with  the  formulae  from  Hicks  (1964).  Table  B-8  shows 
the  output  from  the  EDSTAT-J  computer  program  for  the  full  model 
shown  in  Table  B-5  and  a restricted  model  in  which  the  0 , Bg.  0^, 
coefficients  were  restricted  to  be  0.  This  restriction  tests  the  hypo- 
thesis that  the  interaction  effects  are  all  0.  The  full  model  is: 

Y = p0ut61x1+s2x2*B3x3+B4x1x2+e5x1x3+ef>x2x3+,<1) 

(2) 

the  restricted  model  is:  Y = 1 ^1 3^3  + £ 

Note  that  the  difference  in  ESS  for  the  two  models  is: 

ESS,,  - ESS  = 320 
R F 
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which  i*  the  same  as  the  difference  between  the  Total  Sums  of  Squares 
and  the  Main  Effect  (X^.X^.X^)  Sums  of  Squares  shown  in  Table  B-7. 

TABLE  B-7 

ANALYSIS- OF- VARIANCE  TABLE 


Source 

D.  F. 

S.  S, 

M.  S. 

X1 

1 

1704 

1704 

X2 

1 

320 

320 

X3 

2 

168 

82 

VX2 

1 

9*1 

1 

VX3 

2 

292* 

k 146 

VX3 

2 

18 

9 

WX3 

2 

\) 

. 5 

Total 

11 

2513 

♦Subtotal  = 320 

The  Multiple  Classification  Analysis  (MCA)  Model  and  Procedure: 

The  MCA  model  was  developed  by  the  personnel  at  the 
Institute  for  Social  Research  at  the  University  of  Michigan  to  assess 
the  relative  importance  of  individual  predictor  variables  in  the 
situation  where  the  predictor  variables  were  correlated.  It  is  a 
specialized  categorical  or  binary  regression  model  in  which  the 
elements  of  the  X-matrix  represent  group  membership  of  data  cases 
within  a particular  category  of  the  variables.  Whereas  Ward  (1962) 
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had  pointed  out  that  Hoffman's  (I960)  measure  of  Relative  Weight  was 
meaningless  in  the  case  of  correlated  predictors,  they  sought  a measure 
that  would  reflect  an  adjustment  for  inter  correlation  among  the  variables. 
Toward  this  end  their  procedure  calculates  "adjusted  means"  for  each 
category  of  each  predictor. 

One  difference  in  the  MCA  formulation  than  that  commonly 
used  in  binary  variable  regression  exists  relative  to  the  restriction 
that  is  used  to  insure  a full  rank  X- matrix.  In  the  case  of  a binary 
regression  formulation  where  the  inclusion  of  the  constant  term  causes 
the  X matrix  to  be  less  than  full  rank,  the  model  is  usually  reparame- 
terized so  as  to  include  only  p-  1 categories,  making  the  constant  term 
identical  to  the  mean  response  for  the  omitted  category.  This  has  the 
impact  of  referencing  all  of  the  other  category  "effects"  to  the  mean 
response  for  the  omitted  category  instead  of  the  grand  mean.  In  the 
case  of  MCA,  a full  rank  X- matrix  is  attained  by  restricting  the 
weighted  sum  of  the  category  means  to  zero,  i.  e.  : 


' v 

' • 

v.*»: 


sA‘ 
* . 


ML* 

3 


'N. 


* 


j=i 


n.  a.  = 0 
3 J 


(G) 


The  algorithm  used  to  solve  the  parameters  of  this  model 
follows  an  iterative  procedure  known  as  the  "sweepout"  method 


Adjusted  means  are  estimates  of  what  the  mean  value  for  each 
category  would  have  been  if  the  predictor  vectors  had  been 
orthogonal.  In  this  sense,  MCA  adjusts  for  non-orthogonalities  in 
the  data  (Andrews,  Morgan,  Sonquist,  1967). 
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(Anderson  and  Bancroft,  1952)  wherein  each  of  the  coefficients  of  the 
normal  equations  is  sequentially  adjusted  until  further  adjustments 
are  below  some  predetermined  threshold  level.  In  describing  xhe 
relationship  between  MCA  and  ANOVA,  Andrews,  Sonquist  and  Morgan 
(1967)  describe  MCA  as  a computerized  version  of  ANOVA  with  unequal 
cell  frequencies.  The  major  difference  again  is  in  the  restriction  of 
the  MCA  model,  equation  G,  versus  equation  F of  the  ANOVA  model. 

It  is  pointed  out  that  the  difference  in  this  restriction  does  affect  the 
value  obtained  for  the  "treatment"  effects  in  the  case  of  unequal  cell 
frequencies,  but  does  not  affect  the  reduction  in  the  total  sums  of 
squares.  MCA  does  not  calculate  "F-test"  values  but  does  output 
the  appropriate  sums  of  squares  for  such  calculations. 

The  MCA  output  for  the  numerical  example  is  shown  in 
Table  B-9.  Sonquist  and  Morgan' 3 (1967)  description  of  the  output 
parameters  is  as  follows: 


ij 


*1. 


T). 


e. 


the  deviation  of  the  j'th  category  mean  of  the  i'th 
predictor,  adjusted  for  the  effects  of  the  other 
predictor  s 

the  correlation  coefficient  indicating  the  ability 
ox  a predictor,  as  categorized,  to  explain  variation 
in  the  dependent  variable 

the  proportion  of  the  Total  Sums  of  Squares  ex- 
plainable by  a given  predictor  "i" 
a coefficient  analogous  to  T)  except  based  on  the 
adjusted  means  for  each  category 


A 


-*% 
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(5^  : a measure  of  the  ability  of  a predictor  to  explain 

variation  in  the  dependent  variable  after  adjusting 
for  the  effects  of  the  other  predictors:  Note:  This 
is  not  in  terms  of  percent  of  variation  explained 
and  the  sum  over  all  predictors  is  not  constrained 
to  be  less  than  or  equal  to  unity 

2 

R : the  multiple  correlation  coefficient  which  indicates 
the  proportion  of  variance  in  the  dependent  vari- 
able explained  by  all  of  the  predictors  together 


■ 


' t 


m 


The  net  advantage  of  using  MCA  instead  of  using  a standard 
regression  routine  to  do  binary  regression  appears  to  lie  in  the  mechan- 
ical convenience  of  defining  the  predictor  matrix  and  in  the  fact  that 
the  deviations  of  the  mean  category  responses  for  all  categories  are 
referenced  to  the  grand  mean  as  opposed  to  ' p- 1 ’ categories  values 
being  referenced  to  the  p'th  category  mean. 


* The  Automatic  Interaction  Detection  (AID)  Technique: 

V ; 

The  AID  technique  fulfills  a different  role  in  the  modeling 
process  than  the  previously  discussed  techniques.  AID  is  a "model 
seeking"  technique  and  as  such  does  not  presume  a specific  structure 
to  the  data.  AID  partitions  the  data  set  via  a stepwise  one-way 
vj  analyais-of- variance  procedure  and  results  in  a model  of  the  form 

* n 

Y.  = Z b.  X..  + e. 

* J 3 
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where:  : response  of  the  j'th  case 

: the  group  mean  of  'n1  mutually  exclusive  groups 

: a binary  indicator  of  group  membership 

e^  : the  random  error  of  the  j'th  case  from  its  group 

mean,  b. 

1 

The  unique  feature  of  this  model  lies  in  the  implicit 
application  of  the  restricti  'n* 


I X.  = 1 
1=1  1 


for  all  j 


In  terms  of  Searle’s  (1971)  categorization  of  restrictions,  this 
restriction  is  an  inherent  part  of  the  model  and  results  from  the 
procedure  of  sequential  "de- clustering"  the  data  units  until  "n" 
mutually  exclusive  groups  exists.  This  de- clustering  takes  the  form 
of  a binary  split  of  the  data  set  based  upon  that  division  that  will 
result  in  the  greatest  reduction  in  the  error  sums  of  squares. 
Anderberg  (1971)  aptly  characterized  AID  as  a "monothetic  divisive 
cluster  analysis  technique"  in  that  it  defines  clusters  by  performing 
sequential  binary  splits  on  one  variable  at  a time. 

It  is  important  to  recognize  that  AID  is  nothing  other  than 
a particular  restricted  version  of  the  general  linear  model.  The 
claim  by  the  authors,  Sonquist  and  Morgan  (1964)  and  Sonquisfc  (1970) 
that  the  model  is  different  because  it  does  not  require  "additivity" 
could  better  be  stated  as: 

AID  is  a specific  case  of  the  general  linear  model  in 
which  the  assumption  of  additivity  is  trivially  satis- 
fied due  to  the  restriction  of  1J  X.  =1  for  each  of  the  * j * 
data  units . 1 
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Figure  B-l  shows  the  AID- Tree  for  the  numerical  example 

problem  and  Table  B-10  shows  the  least  squares  estimates  for  the 

equivalent  regression  model.  In  order  to  reparameterize  the  AID 

model  into  an  equivalent  regression  model,  the  vector  for  group  7 

was  omitted.  Therefore,  the  constant  term  of  the  regression  output 

(1)  reflects  the  group  mean  for  group  7 and  the  other  coefficients 

(2,  3,  4,  5)  reflect  the  deviation  of  group  means  of  groups  8,  9,  4,  and 

6 

6 from  the  mean  of  group  7. 

Being  a "model  seeking"  technique,  AID  functions  to 
analyze  and  display  the  data  set  in  such  a way  that  the  latent  charac- 
teristics and  predictor  relationships  of  the  data  set  are  emphasized. 
This  emphasis  is  accomplished  by  the  construction  of  a binary  tree, 
the  branches  of  which  reflect  the  predictors,  and  categories  of  those 
predictors,  that  explain  the  most  variance  in  the  response  variable. 
The  structure  of  this  tree  represents  a visual  presentation  of  the 
structural  relationships  among  the  predictors.  The  analysis  of  such 
trees  and  the  hypothesis  of  models  there  from  is  discussed  extensively 
in  Chapter  IV. 

The  output  of  the  AID  technique  is  a set  of  mutually 
exclusive  clusters  of  similar  data  units  which  best  explains  the 
variance  in  the  data.  Other  statistical  parameters  relative  to  the 
variance  explainable  by  each  variable  in  each  branch  of  the  tree  are 
also  output. 


Note  that  there  is  a factor  of  1 0 difference  in  the  numbers  appearing 
on  the  AID-Tree  and  those  in  the  corresponding  regression  output. 

A restriction  to  integer  numbers  in  AID  required  multiplication  of 
the  criterion  vector  of  the  numerical  problem  by  a factor  of  10. 
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The  objectives  of  the  application  of  this  analysis  technique 
are  different  than  the  objectives  of  those  techniques  in  which  a "true 
model"  is  presumed  and  then  statistics  are  generated  for  the  testing 
of  hypothesis.  Consequently,  there  io  really  no  direct  compairson 
between  it  and  the  other  techniques.  AID  is  best  described  as  a pre- 
dessor  to  the  application  of  these  other  techniques.  However,  AID 
does  result  in  a valid  model  that  is  a restricted  version  of  the  general 
linear  model.  If  the  mutually  exclusive  groups  into  which  AID  divides 
the  data  set  are  found  to  be  valid  within  the  theoretical  context  of  the 
data  being  analysed,  it  can  be  viewed  as  a binary  regression  model 
about  which  various  hypotheses  may  be  tested. 

A computer  listing  and  user's  description  for  the  A1D4UT/ 
AIDTRE  program  is  included  as  Appendix  C of  this  dissertation. 


* " 


Appendix  B:  Exhibits  1 - 1 9 

This  appendix  consists  of  various  forms,  instructions 
and  computer  outputs  used  in  this  research  and  referenced  in  the  body 
of  the  dissertation.  They  are  included  here  as  supplementary  exhibits. 


Exhibit  Title 

1 Original  Application  Form 

2 Initial  Variables  and  Categories 

3 Revised  Variables  and  Categories 

4 Data  for  224  Sample  Applications 

5 Coded  Predictor  Data  Sheet 

6 Field  Test  Instruction  Set  and  Coded  Application 
Form 

7 Data  for  150  Cross-Validation  Applications 

8 AID4UT  Output  for  Judge  #1 

9 AID4UT  Output  for  Judge  # 2 

10  AID4UT  Output  for  Judge  # 3 

11  AID4UT  Output  for  Judge  #4 

12  AID4UT  Output  for  Judge  #5 

13  AID4UT  Output  for  4/5  Vote 

14  AID4UT  Tree  for  Judge  #6 

15  AID4UT  Tree  for  Judge  #7 

16  Hit/Miss  Table--4/5  Policy  with  Credit  Rating 

17  Hit/ Miss  Table--Compositc  Policy  with  Credit 

Rating 

18  Hit/Miss  Table- - Tentative  Approval  without 
Credit  Rating 

19  Hit/Miss  Table--4/5  Policy  (Local  Model)  with 
Credit  Rating 


Pa«e 


Exhibit  1 

ORIGINAL  APPLICATION  FORM 
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LOAN  AmiCATION 

ICt  Cwb  Oxk  Artkit  LIS  or  OtfNr  i m <f, 


TO. 


L>m«r  S>viiia«  Ajnuci*tion 


Dote: . 


I taa  4 

I fcnriby  hr  mnmbmhip  and  a Um  mmdm  U»  Cmrnmrm  Credit  Code  for  tht  purpeeoo  mi  m*  belowi 

Net  Amount  Required  1 Number  of  Mood*  — . 

a)  UamoXtkrn  Non* 

Jdi mm — ■ — 


i U* 


. ytacv  Takvfce 


. Stock  Q UaitU  □ 


~Cl+  Ctok*  ' 


Hmmm  at  xMm 

<**  WM*e—4)  

u w imtfi  wy  No. 


NhhpW  «| 


. U QvJ  Wvict  B*d«r  No.  . 


.Coo.  Sec.  No.  . 


Nw*  m4  Adtace  «|  folet***  No*  LNh*  AopUeoot: 


OP  My  t « Thtlmtm  fc.  £ut2X^S 

lM>Uhi 




nm.4 


Bi 


UrN 


o«*i  Q pm  yam  0 


. pm  upA  □ per  jw*  Q 


mu'a 


b^UytJ  W — 
0)  1 taro  Q fumb^dcmdM 

1 

i.  , ,.  .,  - 


Mm  m4  4i*0»ii 

□ credit  niyfinitim  p*«dk»|  oi  Hu  following  tkmm 

- 3.  

4. 


<4  My  Bonk  k 

(Q  I Ww  jtte  owtotowdlnt  o*  goQopyi 


0-M.ltotoi 

T*  Wk«  1 lilli* 

INmti 

o^*.w 

Ammmm 



a _ 

Tko  ptoCMdt  of  thk  too*  will  bo  lutth 

Q To  ^mm  md  »%** mv  •"  whide  omuIm  MU**  D to  Qtoo*  QwBfa  bciUf  m follows 


(M  hnwwy  i4ennrtco*Mi 
AUnm 


It,  ndeirby 

(4  ftaMol  Um 

0 f>MPi  frirrfy) 


Q 4»*4  cUwd 

□ ffrert  * pwrtfcue  4e*i4_ 
*C*Mt  |»wo*«ty.  AmoumI  i. 


Typed  Hhhwh 

JUcwded  Vol. 

(mm  , ■ ■■ 

H «U  *r-  


<T)  X 


: rfts*  ! Here  *be  -grisa  d.  fumtdbg  w-A 


i mIwt  tKnjueh  e*Mlnc  duuh«  wsml  or  co*<  railed  Wr  u or  pracwt**  end  fwMdlef  oqwnolmi  omanm  thoMch  amp  I 

eemoenr  mchartpod  * traoeect  bu*w»w  * T«*eei 

1.  a Cwp4h  Lift  on  OU*w<«)  ft.  0 Credt*  H eekk  md  Awidenl  on 

C.  o ^ md  MO— dill  «WM«  4.  □ Comprehmr^o  C«**n«* 

□ rbo.  TWt  0 Addirtooel  Cowomoo  1 □ C oo&Man 

t.  a 


I wtf  hwld  tbo  <o8»w4m  powwfooot  ... 

•od  wpon  >e*  * prwtde  w4  (Wfi  mo  far  any  net  hwUdwd  by  mm. 


£)  2 Mir  Mliwtiad  th*t  tbe  wWton  <4  a eortrac**  or  4**Uv,  tcerpubillltfy  4 materiel*  uted  end  work  perf  owned  for  ih*  fiwuk  pm  of* 
MM*  result  oi  this  enplu  itHw  u m*  reepo»*<hlity  ai«4  you  do  pkx  rJ*r*ntr*  mpinul.  wndrtnuohip  m import  'writ  ptfformod  for  mm. 
tmk  pmmm  m*m|  thi*  *po»»<  «l»n  ccrUlto*  the*  t>v  »>«v«  ita(rm«n<«  m»«i«  tor  ihr  purpow  oi  i »4urt*o  tbo  mUiWm  at  cmhl  tn  inn  m4 
dm  m w4.v^ -«bU  intoniMtMMt  boon**  oh  th«  tnMtworthtmi  oi  ih*  prrton  or  pmon  MKTWf  ham  h mam  aratttad  md  tbrt  «*ch  *!*■■* 
t>af»«a*Ur  to  mdi  M.  to  tovtwto  Unto  .tolto.  to)wr.M  ui^tctod. 


UAD  THX  rOHECOENC  STATEMENT 
CAMFVLLT  SOOHJE  SCMNC. 


Exhibit  2 

ORIGINAL  PREDICTOR  VARIABLES  AND  CATEGORIES 


Variable 


Categories 


Age:* 

Marital  Status: 

Local  Family: 

Draft  Status:* 

Telephone: 

Average  Time 
(Residence  and  Job)* 

Residence  Type: 

Type  Employment:* 

Income  (Per  Month): 

Checking  or  Savings:* 

Credit  Habit  (who  he 
owes):* 

Outstanding  Debts:* 

Loan  Amount: 

Loan  Term  (months):* 
Item  Type: 

Equity: 

Past  Credit  Rating: 
Application  Rating: 
Actual  Response: 


< 23,  23-30,  30-50,  50-63,  > 63 
Married,  Divorced,  Single 

No,  Yes 

Eligible,  Ineligible 
Yes,  No 

<2,  2-10,  > 10 

Rent,  Buying,  Own,  Rent  Free 

Unskilled,  Self-employed,  Military, 
Executive,  Skilled 

<$200,  200-400,  400-500.  500-600,  700-800, 
800-1000,  > 1000 

Yes,  No 
Good,  Bad 

< 500,  500-1000,  1000-2000,  2000-3000, 
3000-4000,  > 4000 

< 150,  150-250,  250-400.  400-600,  600-800, 
> 800 

6-12,  15-18,  24,  36 
Necessity,  non-necessity 

< 10%,  10-20%,  > 20% 

Poor,  Unevaluated,  Medium,  Good 
Bad  (0-2.4),  Good  (2.5-5) 

Approved,  Disapproved 


♦ Variables  that  were  later  redefined  or  re-categorized. 


Exhibit  3 

REVISED  PREDICTOR  VARIABLES  AND  CATEGORIES 
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*4 
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■'*6  M % 

r- 

V 
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.MJi 


Variable 

1.  Age: 

2.  Marital  Status: 

3.  Local  Family: 

4*  Telephone: 

5.  Time  on  Job: 

6.  Time  in  Residence: 

?.  Residence  Category: 

8.  Type  Employment: 

9.  Income  (monthly): 

10.  Bank  Account: 

11.  Financial  Reference: 

12.  Loan  Amount: 

13.  Loan  Term: 

14.  Purchase  Type: 

15.  Equity: 

16.  Credit  Rating: 


Categories 

(1)  (2)  (3)  (4)  (5) 

<21,  22-24,  25-30,  31-40,  >40 

(1)  (2)  (3) 

Married,  Single,  Divorced 

a)  (2) 

No,  Yes 

G)  (2) 

No,  Yes 

(1)  (2)  (3) 

<2,  2-5,  >5 

(1)  (2)  (3) 

< 2,  2-5,  > 5 

(1)  (2)  (3)  (4) 

Rent  free,  Renting,  Buying,  Own 

(1)  (2)  (3)  (4) 

Misc. , Unskilled,  Skilled,  Executive 

(1)  (2)  (3)  (4) 

<$400,  $4QQ-$600,  $600-$900.  >$900 

(1)  (2)  (3)  (4) 

None,  Savings,  Checking,  Other 

(1)  (2)  (3) 

Other,  Minor,  Major 

U)  (2)  (3)  (4) 

< $300,  $300-$600,  $600- $900,  >$900 

(1)  (2)  (3) 

1 year,  2 years,  3 years 

(1)  (2) 

Non-necessity,  necessity 

(1)  (2)  (3) 

< 10%,  10-20%,  > 20% 

(1)  (2)  (3)  (4) 

Poor,  Unrated,  Medium,  Good 
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Exhibit  6 

Procedures  for  Lamar  Data  Sheets 

Field  Test  Instruction  Set  and 
Coded  Application  Form 


■'[l 

A.  .:v- 
'' 


During  the  month  of  July  we  will  be  running  a field- 
evaluation  of  a proposed  computerized  credit  evaluation  system.  Based 
on  extensive  analysis  of  credit  applications  previously  submitted  to 
Lamar,  the  attached  application  sheet  and  explanation  of  categories 
have  been  developed. 

After  a regular  application  blank  has  been  filled  out 
completely,  we  ask  that  the  person  who  would  normally  call  in  the 
credit  report  completely  fill  out  the  special  credit  form,  being  sure 
to  put  down  the  date,  time  and  case  number.  The  case  number  should 
also  be  put  in  the  top  left  corner  of  the  original  application  and  an 
entry  made  in  the  notebook.  If  questions  should  arise  relative  to  which 
category  an  applicant  really  belongs  to  for  a given  item,  you  should 
use  your  own  best  judgment  based  on  the  category  definitions  provided 
in  the  attached  sheet. 

These  sheets  will  be  picked  up  every  few  days  by 
Mr.  Gooch,  who  is  doing  the  evaluation. 
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Case  # Date  and  Time  of  Appfc. Salesman  # 

Age:  21  or  less  22-24 25-30  31-40 over  40 

Marital  Status:  Married Divorced Single  _____ 

Local  Family:  No Yes 

Telephone:  No Yes 

Time  on  Job:  less  than  2 years  2-5  yrs.  over  5 years 

Time  at  Residence:  Less  than  2 years 2-5  yrs.  over  5 yrs. 

Residence  Category:  Rent  free Renting Buying Own  Outright 

Type  Employment:  Misc.  Unskilled Skilled Executive 

Bank  Acct.  : None Savings Checking Ck/Sav._ 

Finan.  Ref.  : Other Minor Major 

Income:  Under  $400 4- $600  6-$900  Over  $900 

Amt.  of  Loan:  Under  $300  3- $600  6- $900  Over  $900 

Equity:  10%  or  less 10-20% Over  20% 

Loan  Term:  1 year 2 years 3 years  _____ 

Purchase  type:  non-necessity necessity 

TO  BE  FILLED  OUT  BY  LOAN  OFFICER 

Credit  Rating:  Poor  Uneval. Med. Good 

Decision:  TD Okay Loan  Officer  Initials  ___ 

Date  and  Time  of  Decision __________________________ 

Special  Consideration? 
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Category  Explanations 


1.  Case  it:  A sequential  number  assigned  to  this  application  for 
bookkeeping  purposes. 

2.  Date  and  Time  of  Application:  Date  and  time  application  wovild 
be  called  in  for  Lamar  approval. 

3.  Salesman  #:  Salesman  making  sale. 

4.  Age:  Age  of  applicant. 

5.  Marital  status:  Current  status. 

6.  Local  Family:  Close  family,  mother,  father,  children  living 
within  50  miles  of  Austin.  Yes,  otherwise,  no. 

7.  Telephone:  Self  explanatory. 

8.  Time  on  Job:  Total  time  in  current  job  or  current  job  plus 

Last  previous  job  if  previous  job  was  the  same  type  job. 

9.  Time  in  Residence:  Total  time  in  current  home  or  in  current 

home  and  last  previous  home  in  Austin. 

10.  Residence  Category:  Rent-free--living  with  parents  or 
relatives:  rest  are  self  explanatory. 

11.  Type  Employment: 

Misc.  : Jobs  which  require  minimum  training  or  may  be 

seasonal  or  somewhat  unsteady.  Examples  are  entertainers, 
retired  , jople,  people  on  welfare  or  social  security,  sports 
people,  bellhops,  circus  and  carnival  workers,  dishwashers, 
farm  laborers  (itinerant),  garbage  collectors,  hod  carriers, 
hospital  aids,  janitors,  laundry  workers,  military  E-3  and 
below,  parking  attendants  and  porters.  Include  pensioners, 
or  students  with  an  income  under  $250  monthly. 
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Unskilled:  Tobs  which  require  little  training  but  which  are  steady. 
Examples  are  apprentice  tradesmen,  bartenders,  cab  drivers, 
clerks,  clerk- stenos,  collectors,  cooks,  dancers,  factory 
assemblers,  factory  machine  operators,  hairdressers,  laborers, 
longshoremen,  maintenance  men,  merchant  seaman,  military 
enlisted  E-4  and  E-5,  miners,  moulders,  musicians,  oil  field 
roustabouts,  painters,  paper  hangers,  radio  announcers,  sales- 
man (under  five  years),  service  station  attendants,  truck  drivers 
(short  haul),  typists,  waiters,  warehousemen  and  watchmen. 

Skilled:  Jobs  normally  involving  considerable  training  skill 
or  education.  Supervision  responsibilities  limited.  Examples 
are  insurance  adjusters,  artists,  auto  and  airplane  mechanics, 
bookkeepers,  bricklayers,  bus  drivers,  cabinetmakers,  cable 
splicers,  carpenters,  chefs,  civil  service  workei  s (grade  WB-9 
and  above),  credit  managers,  dental  and  laboratory  technicians, 
die  makers,  draftsmen,  electricians,  engineers,  manufacturer's 
representatives,  firemen,  food  checkers,  glaziers,  heavy  equip- 
ment operator s (cranes,  bulldozers,  etc.),  inspectors,  instru- 
ment repairmen,  insurance  agents,  inter  viewer  s,  lathe  operators, 
lithographers,  machinists,  mechanics,  military  enlisted  E-7 
and  E-6,  policemen,  pressmen,  salesmen  (over  five  years), 
secretaries,  truck  drivers  (long  distance),  barbers  and  carpet 
layers. 

Executive:  Normally  persons  with  line  authority  supervision 
five  or  more  persons.  Examples  are  accountants,  airplane 
pilots  and  navigators  (commercial),  architects,  chemists, 
crew  chiefs,  dentists,  department  heads,  dispatchers,  editors, 
electronic  specialists,  foremen,  head  chefs  or  maitre  d'hotel 
in  large  restaurant  or  hotel,  law  enforcement  officers,  lawyer, 
manager  (branch,  office  of  sales),  military  officers,  E-8  and 
above,  oil  drillers,  physicians,  programmers,  project  engi- 
neers, railroad  engineers  and  conductors,  supervisors, 
superintendents  and  teachers. 

12.  Bank  Account-  Self  explanatory. 

13.  Financial  Reference:  Previous  and  current  creditors  are 
listed  and  the  following  categories  apply: 

Code  3 (major):  Credit  references  include  major  chain  stores 
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and  major  banks  or  other  large  financial  institutions.  References 
do  not  indicate  reliance  on  short-term,  high- interest  financing. 
Example  references  include:  Household  Finance,  AVCO,  GAC, 
Allied  Finance,  Sears,  Wards,  Scarbroughs,  Joske's,  Dial 
Finance,  all  banks,  savings  and  loans  and  credit  unions. 

Code  Z (minor):  Credit  references  include  only  local  stores 

and  businesses.  References  do  not  indicate  reliance  on  short- 
term, high-interest  financing.  Examples  are  Term  Plan, 
American  Finance,  Aetna,  Republic,  local  department  stores, 
jewelry  stores,  oil  companies,  etc. 

Code  1 (other):  Credit  references  include  only  out-of-town 

sources,  no  reference  at  all,  or  heavy  dependence  on  short- 
term, high- interest  financing. 

14.  Income:  Self  explanatory--use  total  monthly  income  listed 
for  husband  and  wife. 

15.  Amount  of  Loan:  Total  amount  of  contract. 

16.  Equity:  Percentage  down. 

17.  Loan  Term:  (0-17  months)  = 1 year,  (18-27  months)  = 2 years, 

(28-36  months)  = 3 years. 

18.  Purchase  Type:  This  tells  whether  major  item  in  contract  is 

necessity  in  a normal  home  or  not.  Examples  are: 

Necessities:  stove,  refrigerator,  washer,  dryer,  air  con- 

ditioner, television  and  dishwasher. 

Non- necessities:  Sewing  machine,  stereo. 

19.  Credit  Rating:  Judged  credit  rating  from  direct  and  Credit 
Bureau  reports. 

20.  Decision:  Approval  or  disapproval. 

21.  Loan  Officer  Initial:  Initials  of  loan  officer  making  decision. 

22.  Date  and  Time  Approval  or  Rejection  sent  to: _____ 

23.  Special  Consideration?  To  be  annotated  if  special  consideration 
such  as  recourse  or  other  unusual  circumstances  affected 
decision  on  this  application. 
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Exhibit  16  — HIT/MISS  TABLE--4/5  POLICY  WITH  CREDIT  RATING 


288 


*»  «—  **  <m  m * » o « -•  **  r 




► h.'  v - 

2 *5 S 

O ( « « 


taJ  fcJ  «««* 

a * © 

- 5S? 

w e » 


u r o 

• I 

*v  9 e u 

« i « 


(j  ar  v 
Jr  « w 
o r#  *- 

d X t 


290 


><*«**•«*  IVIKIK  — NNrt  C 

<H^HHtWM(VIVNNN^NniNIVNNNNNlWNMN  **r 


I U 

w h *<  r-  * « «»  »■«  ^ rt  « ► *•  *«  mw>  • «mm  «*«*«»•«  * 

• « * 4 «•  •*  »»  t-  « • » «•  • * *>  • « ->  -<  — -*  -.—  »*  -*mt  . 


i—  u w — »»  « «■  «r  ► *%  <•  m»  » «k  «rt  • «f  »*  «t  «**  * «*«•««  m k »»•*•»  *»  * ««  m tf»  •*«  *>*•  w «—  — • • «r- 

1 a a * ««»•»»  b « « m « ««««««  ««  m ism  mmA  m mm  n m nmmmm  m m «v  t*  mm  mn  mm  <«m  nt  N *•»■•  — 

! !>u 

u i » « 


k «•  £\  i>*  am  *****  ■*  + *>++ 1 + + ¥-+- p* +- *p- *-+>+■  r- *-+•  m m m m m mm  *t  m m m + •- + + o » + + <+**  + + 

W * « 

£ * * 


m •••♦••►*«‘«-*,**«<****'*****«i«»*«-*-*  •"  w -«  «*« 

•i  a «*««  «•*<•  « «•  «i  ««  • 

■ 

3 « w 

* % * 


o JF5  ...-*.. 

M O I ••••••• 

(I  o«*-  »•••*•« 

M H?  -* 

X ® 


{••patafloaflasaaa^^Mkfxp^^r-^  ****)*«^**>**>  <ttfiM#«tii»A *«<* 
• v »••••»•••••»•••*-•-••  • • • • v «•••••••••••  • ••••••• 


■x-iwr  *M8ftet>a 


291 


a 

trn  o >>«M#rf|  « 4 4 4t  4 & 

W ^ ^ «««  «««  AffnnfirifinmnMMnM^MM«j^MWiNWft>- 

• 5 

© ta. 

*- 

I o 

'H  £ <*  *>  — *1  *"*"1  + * * *t  *#  ** 

»«  ^ ««  « ««  «nn>  «»«*««««  n««««a  ««n  n nn  nmn  nn^nMMNMMNNN«< 

is  

& * 

**  _ *-’>-'09««a9r>r"?'«<*ir>irct«NMoo««i»-,4)«.*^i*^.4mpi  rin  ri  »^4  *«^4nf*nN  >•« 

■P  Q + *4*444  444  444  4 44  444  4 44*nr4rir,ryr*t**sr*m**i  n««  xirnfl  MN 

It 

v""  **«***•»•*  "•"  — — --o«o#  4 «(4f  V«n««IMN«l 

9 <3  «*  mm  « •*  *r*  **  *■»«  npnnn^nftMMNMh^MtiNikMMw--Hf.w»«^^ 

HS 

m «*• 

WK 

ac  qj 

r * * 
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APPENDIX  C 


Users  Description  and  Listing  for 
A1D4UT/AIDTRE  and  Ancillary  Programs 


Introduction: 

AID4UT/AIDTRE  are  a pair  of  computer  programs  which 
implement  the  Automatic  Interaction  Detection  (AID)  Algorithm  on  the 
University  of  Texas  CDC-6600  computer.  They  are  adaptations  of  the 
AID-4  computer  program  developed  by  the  Personnel  Research  Division 
of  the  Air  Force  Human  Resources  Laboratory,  (AFHRL)  Lackland  AFB, 
Texas. 


The  basic  AID  algorithm  was  developed  in  1964  by  John  A. 
Sonquist 'and  James  N.  Morgan  at  the  University  of  Michigan.  They 
developed  and  exported  the  AID- 2 program  to  various  universities 
(including  UT).  The  AID-4  version  of  the  program  from  which  the 
AID4UT/AIDTRE  programs  were  developed  represents  an  expansion  of 
the  AID- 2 program  by  the  AFHRL  staff ’to  include  the  following  optional 
capabilities: 


1.  An  option  to  select  randomly  a sample  from  the  original 
group  of  observations  and  to  process  only  that  sample. 

2.  An  option  to  select  randomly  a sample  A and  a sample 
B from  the  original  group,  to  process  sample  A first  and  maintain  a 
history  of  the  splits,  and  then  to  force  sample  B to  make  the  same  splits 
as  taken  by  sample  A;  i.  e.  , single  cross-validation  by  forced  splitting. 

3.  An  option  to  do  the  same  as  indicated  in  item  2,  but  in 
addition,  to  process  subsequently  sample  5 freely,  and  then  to  force 
Sample  A to  make  the  same  splits  as  taken  by  sample  B;  i.  e,  , double 
cross- validation  by  forced  splitting. 

4.  An  option  to  allow  sample  A and  sample  B to  be  provided 
by  the  user;  i.  e.  , no  random  sampling  with  AID-4,  and  to  provide  for  single 
cross-validation.  Actually  double  cross-validation  can  be  accomplished 

by  two  separate  runs  and  the  switching  of  samples  A and  B. 

5.  An  opt’"on  to  exclude  cases  with  oeL-of  range  predictor 


values. 
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6.  An  option  to  specify  the  number  of  iterations  «o  he 
printed  in  detail. 

7.  An  option  to  print  only  a brief  summary  of  the  splitting 

involved, 

8.  An  option  to  print  at  each  iteration  each  of  the  following: 

a.  At  value  testing  the  significance  of  the  difference  of 
the  means  of  the  resultant  groups  of  a split. 

2 

b.  A R value  at  each  split  indicating  the  percentage 
of  variance  explained  up  through  the  current  iteration. 

2 c.  A F value  testing  the  significance  of  the  increase 

in  R from  the  previous  iteration. 

d.  A F value  for  a one-way  analysis  of  variance 
conside  ing  all  final  groups  at  this  iteration. 

9.  Various  format  changes  for  control  cards,  reports, 
etc.  , including  additional  messages  explaining  the  direction  of  the  split. 

10.  An  option  to  plot  the  resultant  splits. 

The  AID-4  program  was  operable  on  an  IBM  7040  computer 
system.  It  was  documented  for  use  by  the  AFHRL  in  an  internal  techni- 
cal report. 


Modifications  to  the  AID-4  program  required  for  its  imple- 
mentation on  the  U.  T.  CDC-6600  included: 

a.  Development  of  a Fortran  IV  version  of  the  plot  routines. 

b.  Conversion  from  the  program  linkage  used  on  the  IBM 
7040  into  two  separate  programs  with  a tape  file  interface. 

Further  modifications  were  made  to  improve  the  utility  of 
the  program  and  inciuded: 

a.  Addition  of  an  optional  skeleton  tree  plot  that  includes 
only  the  first  eight  plot  levels. 

b.  Addition  of  a routine  to  display  BSS/TSS  profiles  and 

to  calculate  a coefficient  of  Potential  Explanatory  Power  (PEP)  as 
proposed  by  B.  M.  Finifter  of  Michigan  State  University. 


$ 
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Purpone: 


The  purpose  of  this  appendix  is  to  provide  a User's  descrip- 
tion of  the  AED4UT/ALD TRE  program  and  to  provide  information  on  the 
ancillary  programs  used  with  AID4UT/AIDTRE  in  the  process  of 
Policy  Capturing, 

With  the  express  permission  of  the  Commander,  AFHRL, 
Brooks  AFB,  Texas,  the  description  of  the  input  to  the  AID-4  program 
has  been  taken  directly  from  the  AFHRL  technical  report  entitled 
"Automatic  Interaction  Detection,  AID-4",  July,  1971. 

Additional  information  has  been  added  to  reflect  the  modifi- 
cations made  in  the  AID4UT/ALDTRE  version  of  the  program. 
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THE  AID4UT  ALGORITHM 


The  basic  steps  of  the  AID4UT  algorithm  for  the  "normal 
splitting"  of  groups  into  subgroups;  i.  e.  , no  forced  splitting,  can  be 
summarized  below.  The  only  additional  logic  required  for  forced  splitting 
is  concerned  with  the  generation  of  a history  file  of  normal  splits  to  be 
imposed  upon  a different  set  of  observations. 

Initialization.  Steps  1-4. 

1.  Read  in  all  control  cards  (title  card,  format  card  or 
cards,  description  card,  predictor  cards,  and  criterion  card)  which 
define  the  predictor  variables,  the  criterion  variable,  the  stop  criteria, 
etc.  , and  establish  ail  of  the  options  to  be  processed.  If  an  end-of-job 
card  is  read,  terminate  the  job. 

2.  Print  a summary  of  the  control  cards  and  all  the  vari- 
able definitions  under  "Control  Card  Parameters.  " 

3.  Read  in  the  original  inpi't  file,  according  to  the  options 
specified,  and  generate  the  recode  categories  (classes,  responses, 
codes)  for  each  case  (observation).  Edit  each  case  and  print  each  case 
with  an  out-of-range  predictor  values  are  to  be  excluded,  also  print 

any  such  cases  under  the  same  report.  All  cases  listed  under  this  report 
are  eliminated  at  this  point. 

4.  Identify  all  the  remaining  cases  (assuming  that  no  forced 
splitting  or  random  sampling  or  partial  use  of  the  original  input  file  is 
called  for)  as  belonging  to  group  number  one.  Group  number  one  is  the 
current  candidate  group  for  splitting  and,  indeed  at  the  start,  the  only 
one.  Continue  to  Step  8. 

Test  for  Termination  of  the  Algorithm.  Step  5. 

5.  If  either  the  current  number  of  final  groups  (includes 
the  candidate  groups  and  the  groups  that  cannot  be  split)  equals  MAXGP 
(a  maximum  number  read  in  from  the  description  card),  or  the  number 
of  candidate  groups  equals  90,  terminate  the  algorithm  by  going  to 
Step  25. 

Determine  Which  Group  Should  be  Selected  for  Attempted  Partitioning. 
Steps  6-8. 


6. 


Considering  all  groups  obtained  so  far,  select  one  of 
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them,  group  i,  such  that 

a.  The  total  sum  of  squares  of  that  group  (TSS.)  is 
greater  than  or  equal  to  Pi  (percentage  read  in  from  description  cL:d) 
percentage  of  the  original  total  sum  of  squares  (TSS,^,);  i.  e.  , group 
number  one. 


b.  The  number  of  cases  in  group  i is  not  less  than 
twice  the  value  of  NMIN(the  minimum  number  of  cases  required  to  be  in 
both  resultant  groups  for  a candidate  group  to  be  split,  read  in  from 
description  card). 

c.  The  group  i has  not  already  been  split  into  two 

other  groups. 


group  i. 


d.  There  has  been  no  previous  failure  to  split  the 


e.  The  total  sum  of  squares  of  group  i is  not  leas 
than  the  sum  of  squares  for  any  other  group  that  meets  the  above  four 
criteria  (a-d). 


7.  If  there  is  no  such  group  i,  terminate  the  algorithm  by 
going  to  Step  26. 


8.  The  group  i selected  is  the  current  candidate  group  i 
which  will  be  used  for  an  attempted  splitting  (partitioning).  Identify  it 
with  its  group  number  (i)  and  print 

N.,  W.,  T.  Y.  , £Y.2,  TSS. , and  Y.,  and  S.  D.  ,. 
11111  l i 


Partition  Scan  Over  All  Predictors.  Steps  9-14. 

9.  Set  j,  the  predictor  index,  equal  to  one,  BSS,^  , 
the  maximum  between  sum  of  squares  over  all  predictors  of 
group  i,  equal  to  zero,  and  skip  to  Step  11. 

10.  Increment  j by  1.  If  j is  greater  than  the  total  number 
of  predictors,  go  to  Step  23,  for  the  partition  scan  is  complete.  Other- 
wise, continue  to  Step  11. 


For  each  category  k of  predictor  j of  group  i, 


compute 


N 


W. 


ijk*  ijk' 


IY...  , E Y,  , Y...  . S.  D. ..  . 

ijk  ijk  ijk  ijk 


11. 
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12.  If  there  are  at  least  two  categories  k such  that  N 
continue  to  Step  13.  Otherwise,  print  an  appropriate  message  and  J 
return  to  Step  10,  for  predictor  j has  only  one  category. 


4 0. 


13.  If  predictor  j has  been  defined  as  monotonic,  skip 
Step  14  (do  not  sort  the  Step  11  statistics),  and  go  to  Step  15  instead. 


14.  Sort  the  statistics  produced  in  Step  11  together  with  the 
category  identifiers  for  predictor  j,  into  ascending  order  of  the  means 

<Va)- 

Partition  Scan  Over  the  Categories  of  Predictor  j.  Steps  15-19. 

15.  Set  p,  the  category  index,  equal  to  zero,  BSS, 

, • , , . kMAX 

the  maximum  between  sum  of  squares  over  all  categories  of 

predictor  j,  equal  to  zero,  and  skip  to  Step  17. 


16.  Increase  p by  1 . If  p is  as  large  as  k.,  where  k.  is 

the  number  of  categories  within  predictor  j,  then  print  the  statistics  for 
category  k.  and  go  to  Step  20,  as  all  possible  feasible  splits  to  be  examined 

are  completed. 

17.  If  either 


P 

T N..,  < NMIN  or 

k=0  ,jk 


p 

(N,  - E N,  ) < NMIN, 
1 k=0 


go  back  to  Step  1 6,  as  this  split  cannot  be  made.  Otherwise,  compute 

BSS  , the  between  sum  of  squares  for  the  attempted  split  of  group  i on 
P 

predictor  j between  the  categories  (0,  l,  ....  p)  and  the  adjacent 
categories  (p+1,  p+2,  ...,  k.),  and  print  the  statistics  for  category  p 

according  to  the  print  options  specified. 

18.  If  the  BSS  computed  in  Step  17  is  less  than  BSS,  , 
p kMAX 

the  current  maximum  between  sum  of  squares,  return  to  Step  16. 

Otherwise,  continue  to  Step  19. 
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19.  The  BSSp  computed  in  Step  17  ia  the  largest  encountered 


ao  far  for  predictor  j. 
16. 


Set  BSS.  ...  = BSS  , 

kMAX  p 


store  p,  and  return  to  Step 


Determine  the  Best  Predictor.  Steps  20-21 . 


20.  If  BSS,  the  maximum  between  sum  of  squares 

kMAX  ^ 

over  all  categories  of  predictor  j,  is  equal  to  or  larger  than  jvLA.X* 

the  maximum  between  sum  of  squares  computed  for  all  previously 
scanned  predictors,  continue  to  Step  21.  Otherwise,  return  to  Step  10. 


21. 


13£>£>kMAX  for  predictor  j identifies  the  best  predictor 


and  category  tested  so  far  of  group  i.  Set  BSS.W.^  = BSS,  and 

jMAX  kMAX 

return  to  Step  10. 

Determine  Whether  to  Make  Split,  Steps  23-24. 

23.  If  the  maximum  between  sum  of  square.,  BSS.W._,, 

j MAX 

obtained  after  the  scan  of  each  category  of  each  predictor  of  candidate 
group  i is  equal  to  or  greater  than  P2(per centage  read  in  from  descrip- 
tion card)  percentage  of  the  original  total  sum  of  squares,  TSS^, 

continue  to  Step  24.  If  not,  eliminate  group  i as  having  failed  in  a split 
attempt,  compute  residuals  as  the  options  specify,  and  return  to  Step  6. 

24.  Split  group  i into  two  new  groups,  group  NCF+1  and 
group  NCF+2,  where  NCF  is  the  current  number  of  candidate  and  final 
groups,  according  to  the  predictor  and  category  identified  in  Step  21. 
Print  all  information  concerning  the  split  including  BSS 


BSSiMAX/TSSr 


jMAX’ 

and  the  t value  testing  the  significant  difference  between 


the  means  of  group  NCF+1  and  group  NCF+2.  In  addition,  print  a "Current 
Summary"  indicating  at  this  point  the  total  number  of  candidate  groups 
(including  group  NCF+1  and  group  NCF+2)  and  groups  that  cannot  be  split, 


TSSt,  B5St,  WSSt,  R‘ 


the  tot  1 proportion  of  the  variance  explained, 


an  F value  to  test  the  significance  of  the  reduction  of  the  error  sum  of 
squares  due  to  the  new  split,  and  an  F value  for  a one-way  analysis  of 
variance  for  all  final  groups  so  far.  Add  group  NCF-U  and  group  NCF+2 
to  the  number  of  candidate  groups  ar.d  list  all  candidate  groups,  according 
to  print  options  specified,  before  returning  to  Step  5. 


Termination  of  the  Algorithm.  Steps  25-27. 


25.  Either  the  current  number  of  final  groups  equals  MAXGP 
or  the  number  of  candidate  groups  equals  90.  Print  an  appropriate  message, 
terminate  the  algorithm,  and  go  to  Step  27. 

26.  There  are  no  more  groups  eligible  for  attempted  parti- 
tioning. Print  an  appropriate  message  and  go  to  Step  27. 

27.  Compute  any  residuals,  according  to  options  specified 
and  return  to  Step  1 . 


The  Program: 


The  compiled  AID4UT/ AID TRE  programs  currently  exists 
on  Permanent  File  4409  at  the  UT  Computational  Center.  The  AID4UT 
program  processes  the  data  set  and  determines  the  binary  splits  that 
best  explain  the  variability  in  the  data.  On  option,  it  prints  out  the 
results  for  each  iteration  and  prints  or  punches  a residuals  list  corres- 
ponding to  the  final  groups.  On  option,  it  produces  tape  files  for  use 
by  the  A1DTRE  program  in  graphically  displaying  the  skeleton  and  detail- 
ed AID- Trees.  The  AIDTRE  program  operates  in  tandem  with  the 
AID4UT  program. 


Input  Data: 


The  input  for  the  ALD4UT/AIDTRE  progratns  consists  of 
system  control  cards,  program  control  cards  and  source  data  (either  on 
cards  or  on  tape).  An  example  of  a deck  setup  is  shown  in  Figure  C-l. 

In  the  itemized  description  of  the  input  data,  a symbol  of 
"R",  "O",  or  is  given  by  each  field.  These  symbols  mean: 

R - field  required 
O = field  optional 

* = exception  described  in  field  description. 
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ITEMIZED  INPUT  DESCRIPTION 
AID- 4 TITLE  CARD 


Card 

Columns  Description 


1 


2-49 


50 


R The  card  type,  punched  equal  to  1,  which  identifies  the 
title  card  and  initiates  each  AID-4  run.  A title  card  is 
required  at  the  beginning  of  each  run,  and  if  not,  a termi- 
nal error  will  occur. 

AID-4  runs  can  be  "stacked,  " i,  e.  , more  than  one  run  made 
within  the  same  job,  only  under  the  following  conditions: 

a.  Each  run  in  the  stack  does  not  require  an  "unacces- 
sible  file"  such  as  a tape  file  that  is  not  currently 
mounted  on  a tape  drive  or  such  as  a tape  file  that 
cannot  be  correctly  positioned  with  the  rewind  option. 

b.  The  runs  do  not  contain  conflicting  options  such  as  an 
IRUN  equal  to  3. 

O The  title  which  is  printed  at  the  beginning  of  the  output 
under  "Control  Card  Parameters."  Forty-eight  alpha- 
numeric characters  are  provided  for  the  user  to  identify 
a particular  run. 

R IRUN,  the  run  option,  which  can  be  summarized  as  the 
following: 


Value  of  IRUN 


Meaning 


0 Use  every  case  in  the  original 
input  file  for  a normal  AID-4  run; 
i.  e.  , no  forced  splitting,  provide 
ESS/ TSS  profiles. 

1 Select  a random  sample  A from  the 
original  input  file,  and  use  only  the 
cases  that  belong  to  A for  a normal 
AID-4  run.  Then  force  those  cases 
that  belong  to  E to  make  the  same 
splits  as  taken  by  A;  i.  e.  , single 
cross-validation. 
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3 Select  a random  sample  A and  a 
random  sample  B from  the  original 
input  file,  and  use  only  the  cases 
that  belong  to  A for  a normal  AID-4 
run.  Then  force  those  cases  that 
belong  to  B to  make  the  same  splits 
as  taken  by  A.  Then  use  only  the 
cases  that  belong  to  B for  a normal 
AID-4  run,  and  force  those  cases 
that  belong  to  A to  make  the  same 
splits  as  taken  Dy  B;  i.  e.  , double 
cross-  validation. 

4 Given  sample  A and  sample  B,  use 
only  the  cases  that  belong  to  A for  a 
normal  AID  run.  Then  force  those 
cases  that  belong  o B to  make  the 
same  splits  as  taken  by  A.  Note 
that  double  cross-validation  can  be 
accomplished  by  submitting  a second 
job  with  A and  B switched. 

51-56  R*  NCPERM,  the  number  of  cases  in  the  original  input  file. 

NCPERM  is  optional  if  IRUN  equals  zero  and  the  input 
file  is  a tape  file  ending  with  an  end-of-file  mark;  i.  e.  , 
NCPERM  can  be  left  blank,  in  which  case  all  records  in 
the  input  file  will  be  processed  until  an  end-of-file  condi- 
tion is  reached. 

NCPERM  is  also  optional  if  IRUN  equals  zero  and  the 
input  file  is  a card  file  providing  an  "end-of-file  card" 
is  placed  behind  the  last  data  card. 

NCPERM  is  not  applicable  if  IRUN  equals  four. 

57-58  0 NREELS,  the  number  of  reels  for  NCPERM. 

If  NREELS  equals  blanks  or  zeros,  NREELS  is  assumed 
to  be  one. 

If  IRUN  = 4,  NREELS  for  sample  A and  B must  be  equal. 
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59-64  R*  NSAMA,  the  number  of  cases  to  be  randomly  selected  in 
sample  A. 

NSAMA  is  optional  if  1RUN  equals  four;  i.  e.  , NSAMA  can 
be  left  blank  in  which  case  all  cases  in  the  sample  A will 
be  processed  until  an  end-of-file  condition  is  reached. 

NSAMA  is  not  applicable  if  IRUN  equals  zero. 

65-70  R*  NSAMB,  the  number  of  cases  to  be  randomly  selected  in 
sample  B. 

NSAMB  is  optional  if  IRUN  equals  four;  i.  e.  , NSAMB  can 
be  left  blank  in  which  case  all  cases  in  the  sample  B will 
be  processed  until  an  end-of-file  condition  is  reached. 


NSAMB  is  not  applicable  if  IRUN  eqi  \ls  zero  or  one. 

71-78  R*  NUM,  an  eight  digit  starting  random  number  to  be  used  in 
the  random  selection  process  when  IRUN  equals  one,  two, 
or  three.  NUM  should  not  be  a multiple  of  17  or  5882353. 

NUM  is  not  applicable  if  IRUN  equals  zero  or  four. 

79-80  R IFMT,  the  number  of  variable  format  cards  to  be  read  in 
next.  The  range  of  IFMT  is  1 < IFMT  < 4. 


Card 

Columns 

1-78  R 
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AIR-4  FORMAT  CARD  OR  CARDS 


Description 

The  variable  format  beginning  with  a left  parenthesis  and 
ending  with  a right  parenthesis  specifies  the  integer  type  (I) 
conversion  of  the  numerical  data  for  the  input  file  or  files. 

Only  integer  type  (I)  fields  can  be  defined  within  the  parenthesest 
with  X conversion  to  skip  characters  and  the  slash  {/)  to  indi- 
cate the  beginning  of  new  records. 

The  maximum  number  of  input  variables  to  be  read  in  (as 
specified  by  NV  in  the  description  card)  is  83  (80  fields  for 
predictors,  one  field  for  the  criterion,  one  field  for  identi- 
fication, and  one  field  for  weights).  Each  predictor  field 
read  in,  however,  need  not  be  used.  Therefore,  for  certain 
sequences  of  runs,  it  may  be  advantageous  to  use  the  same 
format  for  all  runs  and  control  predictor  selection  with  the 
predictor  cards  only. 

The  only  permissible  characters  in  the  input  files  to  be  read 
by  the  integer  (I)  type  format  are  (J  - 9,  +,  -,  and  blank. 
Otherwise,  a FORTRAN  terminal  error  will  occur. 

The  number  of  format  cards  expected  by  AID-4  depends  on 
the  value  of  IFMT  in  the  title  card.  At  least  one  format  card 
is  required  for  each  run  with  a maximum  of  no  more  than  four. 

For  IRUN  equal  to  zero  through  three,  the  variable  format  card 
(or  cards)  specifies  the  arrangement  of  the  data  in  the  original 
input  file. 

For  IRUN  equal  to  four,  the  variable  format  card  (or  cards) 
specifies  both  the  given  sample  A and  the  given  sample  B. 
Therefore,  the  arrangement  of  sample  A and  sample  B must 
be  identical. 
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Not  used. 
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AID- 4 DESCRIPTION  CARD 


Description 


The  card  type,  punched  equal  to  3,  which  identifies  the 
description  card.  A description  card  is  required  for  each 
AID-4  run. 


Pi,  the  percentage  of  the  original  total  sum  of  squares  that 
must  be  contained  in  any  given  group  if  that  group  is  to  be- 
come a candidate  group  for  splitting.  That  is,  the  total 
sum  of  squares  in  any  given  group  must  be  greater  than  or 
equal  to  Pi  percent  of  the  original  total  sum  of  squares  for 
that  group  to  become  a candidate  group.  Five  decimal  places 
are  assumed  (read  with  an  F5.  5 format).  For  example,  to 
specify  one  percent,  the  user  would  punch  01000  in  columns 
2-6.  Note  that  the  decimal  point  is  not  actually  punched  in 
the  card.  The  range  of  Pi  is  .00001  s Pi  < 1.  Pi  punched 
equal  to  00001  essentially  deactivates  this  stopping  criterion. 
(Cannot  be  left  blank.  ) 


P2,  the  best  split  of  a given  candidate  group  must  reduce  the 
original  within  (unexplained,  error)  sum  of  squares  by  PZ 
percentage  of  the  original  total  sum  of  squares  if  that  group 
is  to  be  split.  That  is,  the  largest  between  sum  of  squares 
obtained  after  the  scanning  of  all  predictors  of  a given 
candidate  group  must  be  equal  to  or  greater  than  P2  percent- 
age of  the  original  total  sum  of  squares  for  that  candidate 
group  to  be  split.  Furthermore,  once  a candidate  group 
has  filled  the  F2  requirement,  that  group  will  not  ke  consi- 
dered for  splitting  again,  even  though  the  group  meets  the  P 
requirement.  The  range  of  P2  is  such  that  P2  £ Pi.  Ordi- 
narily F2  is  set  equal  to  PI.  (Cannot  be  left  blank.  ) 


Note  that  Pi  and  P2  define  an  upper  bound  on  the  number  of 
possible  splits. 


MAXGP,  the  maximum  number  of  final  groups  (includes 
candidate  group?  and  groups  that  cannot  be  split),  regardless 
of  the  values  of  Pi  and  P2,  that  can  be  obtained  before  the 
algorithm  will  terminate. 
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If  forced  splitting  is  called  for,  i.  e.  , IRUN  greater  than 
one,  the  maximum  value  for  MAXGP  is  90. 

Ordinarily  MAXGP  is  set  high  enougn  so  that  the  stopping 
criterion  is  determined  by  Pi,  P2,  and  NMIN. 

17-21  R NMIN,  the  minimum  number  oi'  cases  required  to  be  in  both 
resultant  groups  for  a candidate  group  to  be  split.  NMIN 
must  be  2 2 or  a fatal  error  will  result. 

22-26  R#  KSTOP,  the  number  of  iterations  to  be  printed  in  detail,  i.  e.  , 
the  details  of  the  scanning  of  ail  predictors  for  the  candidate 
groups.  After  KSTOP  is  exceeded,  only  summary  informa- 
tion is  printed. 

If  KSTOP  is  left  blank,  KSTOP  will  be  set  to  zero  and  only 
summary  information  will  be  printed. 

If  KSTOP  equals  -1,  only  a "Split  Summary"  is  printed. 
(Minimum  output) 

27-29  R*  NV,  the  number  of  variables  to  be  read  in  from  the  input 

file.  This  number  includes  the  predictor  variables,  criterion 
variable,  the  identification  variable  (if  applicable),  and  the 
weight  variable  (if  applicable).  NV  is  required,  must  be  less 
than  or  equal  to  83,  and  must  agree  with  the  format  as  speci- 
fied in  the  format  cards . 

30-32  R*  NR,  the  field  number  of  the  case  identification  variable  as 
determined  from  the  format  card  or  cards. 

If  residuals  are  called  for  (IRESID  greater  than  zero),  NR 
must  be  specified  (0  < NR  s 83).  Otherwise,  NR  can  be  left 
blank. 

33  0 KRW,  the  rewind  option  which  determines  whether  or  not  to 

rewind  the  original  input  file  (a  tape  file)  before  the  pro- 
ces  *ing  for  a given  run  begins. 

If  KRW  is  equal  to  a zero  or  a blank,  the  original  input  file 
is  not  to  be  rewound. 

If  KRW  is  equal  to  or.e,  the  file  is  to  be  rewound. 
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If  the  original  inpul  file  is  a card  file  {ICARD  equal  to  one), 
KRW  is  not  applicable. 

34  R IOPT,  the  exclude  option  which  determines  whether  or  not 

cases  with  out- of- range  predictor  values,  as  defined  by  the 
predictor  cards,  are  excluded. 

If  IOPT  is  equal  to  a blank  or  a zero,  the  cases  with  out-of- 
range  predictor  values  are  not  excluded.  Predictor  values 
less  than  the  specified  minimum  values  are  assigned  recode 
category  00  and  predictor  values  greater  than  the  specified 
maximum  values  are  assigned  the  largest  recode  categories. 

If  IOPT  is  equal  to  one,  the  cases  with  out-of-range  predictor 
values  ?re  excluded.  The  excluded  cases  are  printed  in 
the  "Reject  Summary"  if  the  summary  is  called  for;  i.  e.  , 
KREJ  equal  to  one  (see  KREJ  description). 

IOPT  is  extremely  important  in  that  IOPT  determines  the 
format  of  the  predictor  cards  (see  description  of  AID-4 
predictor  cards). 

35  0 KREJ,  the  reject  summary  print  option  which  determines 

whether  or  not  the  "Reject  Summary"  is  printed.  The 
summary  lists  all  those  out-of- range  cases  that  are 
excluded  because  of  invalid  predictor  values  or  invalid 
criterion  values. 

If  KREJ  is  a zero  or  a blank,  the  summary  is  not  printed. 

If  KREJ  is  one,  the  summary  is  printed. 

36  0 IRESID,  the  residual  summary  option  which  determines 

whether  or  not  residuals  (the  criterion  values  minus  the 
final  group  mean)  are  computed. 

If  IRESID  is  a zero  or  a blank,  no  residuals  are  computed. 

If  IRESID  is  one,  the  residuals  are  printed  in  the  "Residual 
and/or  Reject  Summary,  " This  summary  is  printed  each 
time  a final  group  is  obtained;  i.  e. , a group  that  cannot  be 
split  any  more.  Approximately  200  residuals  are  printed 
per  page. 


If  IRESID  is  two,  the  residuals  are  punched  into  cards.  Also 
IIIUN  must  be  less  than  two. 

If  IRESID  is  three,  the  residuals  are  written  on  Tape  8. 

Also  IRUN  must  be  less  than  two.  With  this  option  Tape  8 
can  be  rewound  and  printed  at  job  completion  to  give  a 
concise  list  of  residuals. 

ICARD,  the  card  input  option  which  determines  whether  or 
not  the  original  input  file  is  cards. 

If  ICARD  is  zero  or  blank,  the  original  input  file  i3  presumed 
to  have  been  prepositioned  on  Tape  25. 

If  ICARD  is  one,  the  file  is  cards.  However,  IRUN  must  be 
less  than  three. 

ITREE,  the  plot  option  which  determines  whether  or  not  the 
normal  splitting  phases  (1  and  3)  are  plotted.  The  forced 
splitting  phases  (2  and  4)  cannot  be  plotted.  If  ITREE  a one, 
Tape  10  and  Tape  14  are  generated  for  use  by  AIDTRE. 

If  ITREE  is  zero  or  blank,  no  tapes  are  generated  it  ITREE 
is  zero  or  blank,  no  tapes  are  generated. 

If  ITREE  is  one,  only  the  detailed  I ID- TREE  will  be  generated. 

If  ITREE  is  two,  detailed  and  skeleton  AID- TREES  will  be 
generated. 

NOGO,  the  control  card  checking  option  which  determines 
whether  or  not  the  run  terminates  after  the  printing  of  all 
control  cards. 

If  NOGO  is  a zero  or  blank  normal  pr  icessing  is  assumed. 

If  NOGO  is  one,  the  run  will  terminate  after  the  printing  of 
the  control  card  summary. 

NOGO  runs  do  not  require  any  files  and  can  be  stacked  with 
other  runs. 

IESS,  the  error  sum  of  squares  option  which  determines 
whether  or  not  the  error  sum  of  squares  for  forced  splits  is 
printed. 
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If  IESS  is  a zero  or  a blank,  the  error  sum  of  squares  is 
W't  printed. 


If  IESS  is  a cne,  the  error  sum  of  squares  is  printed. 


41-80  Not  used. 


#*«#* 
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AID- 4 PREDICTOR  CARDS 


Card 

Column  Description 


1 R The  card  type,  punched  equal  to  4,  which  identifies  the  pre- 

dictor card.  At  least  one  predictor  card  is  required  for  each 
predictor  (specified  categories  can  require  more  than  one 
card). 


2-19  R The  name  of  the  predictor  which  is  printed  at  the  beginning 

of  the  output  under  "Control  Card  Parameters"  and  each  time 
a split  is  made  on  the  predictor  named.  Eighteen  alphanumeric 
characters  are  provided  for  the  name. 

20-22  R The  field  number  of  the  predictor  as  specified  by  the  format 
card  or  cards. 


23  R KBL1,  the  first  predictor  specification  variable,  which 

determines  the  type  of  the  predictor. 

If  KBLl  is  zero,  the  predictor  is  free-floating;  i.  e.  , the 
categories  within  the  predictor  are  not  sorted  and  remain  in 
original  recode  order  when  the  scanning  process  begins. 

If  a predictor  is  free-floating,  the  between  sum  of  squares 
obtained  by  the  scanning  process  will  be  maximized  over 
all  possible  splits  for  that  predictor.  If  a predictor  is  mono- 
tonic, the  between  sum  of  squares  obtained  will  be  maximized 
only  over  the  monotomc  splits  for  that  predictor.  This  does 
not,  however,  imply  that  all  predictors  should  be  typed  as 
free-floating,  for  certain  predictors  make  little  sense  as 
such.  The  type  is  dependent  upon  the  nature  of  the  predictor. 

24  R KBJL2,  the  second  predictor  specification  variable,  which 

determines  how  the  predictor  is  to  be  defined.  The  values 
of  the  predictor  (as  read  by  the  variable  format)  must  be 
positive  or  negative  integers  with  a range  of  -99999  to  999999. 
AID-4  recodes  the  values  of  the  predictor  into  categories 

numbered  uO,  01 39.  The  maximum  recode  value  is 

39;  i.  e.  , the  total  number  of  recode  values  for  any  one 
predictor  cannot  exceed  40.  Two  methods  of  defining  recodes 
are  provided. 
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If  KBL2  is  zero,  the  predictor  is  defined  to  have  equal  intervals- 
i.  e.  , the  recode  categories  are  determined  from  a minimum 
value;  a maximum  value,  and  an  interval  length.  See  descrip- 
tion of  columns  25-80. 

If  KBL2  is  one,  the  predictor  is  defined  to  have  specified 
categories;  i.  e.  , the  categories  are  determined  from  speci- 
fied input  and  recode  values.  See  description  < f columns 
25-80. 

25-80  The  format  of  columns  25-80  is  dependent  upon  KBL2  and  the 

exclude  option,  IOPT  (column  34  of  the  description  card), 
and  can  be  summarized  as  the  following  four  possibilities: 

I.  KBL2  equal  to  zero  and  IOPT  equal  to  a blank  or  zero; 
i.  e.  , equal  intervals  and  cases  with  out-of-range  predictor 
values  not  excluded. 
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I 25-30  R The  minimum  value,  which  determines  the  predictor  values 
to  be  included  in  the  first  category;  i.  e.  , category  00.  Any 
predictor  value  less  than  or  equal  to  the  minimum  value  will 
be  included  in  category  00.  Also  any  predictor  value  less 
than  one  interval  greater  than  the  minimum  value  will  be 
included  in  category  00. 

I 31-36  R The  maximum  value,  which  determines  the  predictor  values 
to  be  included  in  the  last  category.  Any  predictor  value 
greater  than  or  equal  to  the  maximum  value  will  be  included 
in  the  last  category. 

I 37-42  R The  interval  length,  which  together  with  the  minimum  and 

maximum  values,  determines  the  number  of  recode  categories. 
The  number  of  categories  is  computed  by 

((max-min)/inter  val)  + 1, 

which  must  be  less  than  40.  Note  that  remainders  after 
division  are  truncated. 

For  predictor  values  greater  than  the  minimum  value  and 
less  than  the  maximum  value,  the  recode  categories  are 
computed  by 
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(predictor  value  - min) 
interval 

Note  that  remainders  are  truncated. 

I 43-45  R Specified  recode  category  1,  the  value  of  which  is  assigned  to 
any  predictor  value  equal  to  specified  input  value  1 {columns 
I 46-51),  Provision  is  made  for  reassigning  up  to  three  values 
of  the  predictor  (using  specified  recode  categories  1,2,  and  3). 
Note,  however,  that  the  specified  input  values  1,  2,  and  3 
should  be  greater  than  or  equal  to  the  minimum  value  and 
less  than  or  equal  to  the  maximum  value,  as  the  range  check 
for  minimum  and  maximum  values  is  performed  first.  If 
a predictor  value  is  less  than  the  minimum,  the  value  is  set 
equal  to  the  minimum.  If  a predictor  value  is  greater  than 
the  maximum,  the  value  is  set  equal  to  the  maximum. 

If  the  recode  is  not  used,  a -1  must  be  punched  in  columns 
I 44-45. 

I 46-51  R Specified  input  value  1. 

I 52-54  R Specified  recode  category  2,  the  value  of  which  is  assigned  to 
any  predictor  value  equal  to  specified  input  value  2 (columns 
I 55-60). 

If  the  recode  is  not  used,  a -1  must  be  punched  in  columns 
I 53-54. 

I 55-60  R Specified  input  value  2. 

I 61-63  R Specified  recode  category  3,  the  value  of  which  is  assigned 
to  any  predictor  value  equal  to  specified  input  value  3 
(columns  I 64-69). 

If  the  recode  value  is  not  used,  a -1  must  be  punched  in 
columns  I 62-63. 

I 64-69  R Specified  input  value  3. 

70-80  Not  used. 
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II.  KBL.2  equal  to  zero  and  IOPT  equal  to  one;  i.  e.  , equal 
intervals  and  cases  with  out-of-range  predictor  values 
excluded. 

II  25-30  R The  minimum  value  by  which  any  predictor  value  less  than 
the  minimum  value  is  excluded. 

II  31-36  R The  maximum  value  by  which  any  predictor  value  equal  to 
or  greater  than  the  maximum  is  excluded. 

II  37-42  R The  interval  length,  which  together  wit.  the  minimum  and 
maximum  values,  determines  the  number  of  recode  cate- 
gories. The  number  of  categories  is  computed  by 

( (max-  min)  / inter  val) , 

which  must  be  less  than  40.  Note  that  remainders  after 
division  are  truncated. 

For  predictor  values  greater  than  or  equal  to  the  minimum 
value  and  less  than  the  maximum  values  the  recode  cate- 
gories are  computed  by 

(predictor  value  - min) 
interval 


Note  that  remainders  are  truncated. 


II  43-45  R Specified  recode  category  1,  the  value  of  which  is  assigned 
to  any  predictor  value  equal  to  specified  input  value  1 
(columns  II  46-51).  Provision  is  made  for  reassigning 
up  to  three  values  of  the  predictor  (using  specified  recode 
categories  1,  2,  and  3).  Note,  however,  that  the  speci- 
fied input  values  1,  2,  and  3 should  be  greater  than  or 
equal  to  the  minimum  value  and  less  than  the  maximum 


• m 1 a 'i  o 4-U/i.  ■»-  « nr  rt  Ir  rv\  ini 


t r»  i 11 


m 


LliU  i- *■***>! 


values  is  performed  first.  If  a predictor  value  is  less 
than  the  minimum  or  greater  than  or  equal  to  the  maximum, 
the  case  is  excluded. 


If  the  recode  is  not  used,  a -1  must  be  punched  in  columns 
44-45. 


H 46-51  R 
n 52-54  R 


II  55-60  R 
H 61-63  R 


H 64-69  R 
n 70-80  R 

in  25-27  R 

in  28-33  R 
HI  34-36  R 

ni  37-42  R 


Specified  input  value  1.  Note  that  this  value  cannot  be 
greater  than  or  equal  to  the  maximum,  or  less  than  the 
minimum. 

Specified  recode  category  2,  the  value  of  which  is  assigned 
to  any  predictor  value  equal  to  specified  input  value  2 
(columns  II  55-60). 

If  the  recode  is  not  used,  a -1  must  be  punched  in  columns 
II  53-54. 

Specified  input  value  2.  Note  that  the  value  cannot  be 
greater  than  or  equal  to  the  maximum,  or  less  than  the 
minimum. 

Specified  recode  category  3,  the  value  of  which  is  assigned 
to  any  predictor  value  equal  to  specified  input  value  3 
(columns  II  64-69). 

If  the  recode  value  is  not  used,  a -1  must  be  punched  in 
columns  II  62-63, 

Specified  input  value  3.  Note  that  the  value  cannot  be 
greater  than  or  equal  to  the  maximum,  or  less  than  the 
minimum. 

Not  used. 

III.  KBL2  equal  to  one  and  IOPT  equal  to  a blank  or  zero, 
i.  e.  , specified  categories  and  cases  with  out-of-range 
predictor  values  not  excluded. 

Specified  recode  category  1,  the  value  of  which  is  assigned 
to  any  predictor  value  equal  to  or  less  than  specified 
input  value  1 (columns  III  28-33). 

Specified  input  value  1. 

Specified  recode  category  2,  the  value  of  which  is 
assigned  to  any  predictor  value  greater  than  specified 
input  value  1 and  less  than  or  equal  to  specified  input 
value  2 (columns  III  37-42). 

Specified  input  value  2. 
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El  43- 15  R 


Specified  recode  category  3,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  2 
and  les9  than  or  equal  to  specified  input  value  3 (columns 
m 46-51). 


Ill  46-51  R Specified  input  value  3. 


HI  52-54  R 


in  55-60 


HI  61-63  R 


Specified  recode  category  4,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  3 
and  less  than  or  equal  to  specified  input  value  4 (columns 
in  55-60). 

Specified  input  value  4. 

Specified  recode  category  5,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  4 
and  less  than  or  equal  to  specified  input  value  5 (columns 
HI  64-69). 


HI  64-69  R Specified  input  value  5. 
HI  70-80  Not  used. 


If  more  than  five  categories  are  required,  additional  pre- 
dictor cards  can  be  used  by  duplicating  columns  in  1-24 
and  continuing  in  columns  HI  25-69  respectively.  As  many 
cards  per  predictor  as  necessary  can  be  used  as  long  as 
the  maximum  of  40  recode  categories  is  not  exceeded; 
i.  e.  . 00-39. 

The  last  specified  input  value  must  always  equal  999999 
and  any  predictor  value  greater  than  the  previous  speci- 
fied input  value  is  assigned  the  last  specified  recode 
category.  For  example,  if  specified  input  value  4 equaled 
999999.  any  predictor  value  greater  than  input  value  3 
would  be  assigned  recode  category  4. 

IV.  KBL2  equal  to  one  and  IOPT  equal  to  one;  i.  e. , 
specified  categories  and  cases  with  out-of-range  predictor 
values  excluded. 

IV  25-27  R Blank  {except  for  continuation  cards;  see  next  to  last 
paragraph  below. ) 


IV  28-33  R 
IV  34-36  R 

IV  37-42  R 
IV  43-45  R 

IV  46-51  R 
IV  52-54  R 

IV  55-60  R 
IV  61-63  R 

IV  64-69  R 
IV  70-80 


The  lower  bound  by  which  any  predictor  value  less  than  or 
equal  to  the  lower  bound  is  excluded. 

Specified  recode  category  2,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  the  lower  bound  and 
less  than  or  equal  to  specified  input  value  2 (columns  IV 
37-42). 

Specified  input  value  2. 

Specified  recode  category  3,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  2 
and  less  than  or  equal  to  specified  input  value  3 (columns 
IV  46-51). 

Specified  input  value  3. 

Specified  recode  category  4,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  3 
and  less  than  or  equal  to  specified  input  value  4 (columns 
IV  55-60). 

Specified  input  value  4. 

Specified  recode  category  5,  the  value  of  which  is  assigned 
to  any  predictor  value  greater  than  specified  input  value  3 
and  less  than  or  equal  to  specified  input  value  5 (columns 
IV  64-69). 

Specified  input  value  5. 

Not  used. 

If  more  than  three  categories  are  required,  additional  pre- 
dictor cards  can  be  used  by  duplicating  columns  IV  1-24 
and  continuing  in  columns  IV  25-69  respectively.  On 
second  and  following  continuation  cards,  start  next  speci- 
fied recode  category  in  columns  IV  25-27.  As  many  cards 
per  predictor  as  necessary  can  be  used  as  long  as  the 
maximum  of  40  recode  categories  is  not  exceeded;  i,  e.  , 
00-39. 
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AID-4  CRITERION  CARD 


Description 


The  card  type,  punched  eq'ial  to  5,  which  identifies  the 
criterion  card.  One  criterion  card  is  required  for  each 


The  name  of  the  criterion  which  is  printed  at  the  beginning 
of  the  output  under  "Control  Card  Parameters  " Eighteen 
alphanumeric  characters  are  provided  for  the  name. 

The  field  number  of  the  criterion  as  specified  by  the  format 
card  or  cards. 

The  field  number  of  the  weight  as  specified  by  the  format 
card  or  cards. 

If  this  field  is  left  blank,  the  weight  is  set  to  one. 

YMAX,  the  maximum  allowable  value  of  the  criterion 
variable.  If  the  value  of  the  criterion  is  greater  than 
YMAX,  the  case  is  excluded.  The  maximum  value  that 
can  be  specified  for  YMAX  is  999999. 

MD1 , the  first  deletion  value,  by  which  any  criterion  value 
equal  to  MD1  will  be  excluded.  MD1  can  be  deactivated 
by  setting  MDi  greater  than  YMAX. 

MD2,  the  second  deletion  value,  by  which  any  critei  ion 
value  equal  to  MD2  will  be  excluded.  MD2  can  be  deacti- 
vated  by  setting  MD2  greater  than  YMAX. 


49-80 


Not  used. 


AID- 4 END-  Gi-JOD  ^ARD 


Card 

Columns 

1 R 


Description 

The  card  type,  punched  equal  to  9,  which  identified  the 
end-of-job  card  and  terminates  the  job, 

The  purpose  of  the  end- of- job  card  is  to  indicate  the  end 
of  a run  executed  within  a given  job.  The  card  should  be 
thought  of  strictly  as  a control  card  such  as  the  title 
card,  format  card,  etc.  , and  not  as  part  of  the  original 
input  file. 
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AID- 4 LIMITATIONS 


. ./ 


For  the  current  version  of  AID-4,  the  following  list  of  limita- 
tions apply: 


1.  The  maximum  number  of  input  variables  to  be  read  in 
is  83  (80  fields  for  predictors,  one  field  for  the  criterion,  one  field  for 
identification,  and  one  field  for  weights). 

2.  The  maximum  number  of  predictor  variables  is  80. 


3.  The  maximum  number  of  generated  categories  for  all 
predictor  variables  taken  together  must  not  exceed  700.  The  total  number 
of  categories  for  all  predictors,  N^,,  can  be  computed  by  the  following 
formula: 


Nc  = 4NE  + 2Nsr  + 2Nsc 


where  N is  the  number  of  predictor  variables  with  equal  intervals,  N 

£ oK 

is  the  number  of  specified  recode  categories  for  predictors  with  equal 

intervals,  and  N is  the  number  of  specified  recode  categories  for 
SC 

predictors  not  with  equal  categories. 

4.  The  maximum  number  of  criterion  variables  for  any  one 
analysis  is  one. 


5.  The  maximum  number  of  candidate  groups  at  any  given 
iteration  is  90. 


"'<*4- 


jif  ito 


6.  The  maximum  number  of  cards  for  the  variable  input 
format  is  four. 

7.  The  maximum  number  of  final  groups,  including  the 
candidate  groups  and  the  groups  that  cannot  be  split,  is  99999,  the 
maximum  value  of  MAXGP.  Note  that  the  number  of  final  groups  is 
also  dependent  upon  Pi,  P2,  and  NMIN. 


8.  The  range  of  the  recode  categories  is  0 to  39. 

.v 

9.  The  range  of  the  predictor  values  is  -99999  to  999999. 

& 


fx 


TO.  The  range  of  the  criterion  values  is  -99999  to  999999. 

11.  The  range  of  the  weight  values  is  determined  by  the 

iable  format. 


12.  The  minimum  value  for  NIvtIN  is  2. 
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AID- 4 LIST  OF  TERMINAL.  ERRORS 


Error  Number 


Description 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

15 

16 

17 

18 

19 

20 

21-24 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 

36 


AID-4  title  card  (card  type  1)  expected. 

IRUN  out-of-range  (greater  than  4). 

IRUN  equal  to  1 , 2,  or  3 and  NCPERM'equal  to  zero. 

IRUN  equal  to  1 , 2,  or  3 and  NSAMA  equal  to  zero. 

IRUN  equal  to  2 or  3 and  NSAMB  equal  to  zero. 

IRUN  equal  to  1 , 2,  or  3 and  NUM  equal  to  zero. 

IRUN  equal  to  1 , 2,  or  3 and  NUM  is  a multiple  of  17. 

IRUN  equal  to  1 , 2,  or  3 and  NUM  is  a multiple  of 

5882353. 

IFMT  out-of- range  (equal  to  zero  or  greater  than  4). 
AID-4  description  card  (card  type  3)  expected. 

Pi  out-of-range  (greater  than  1). 

P2  out-of- range  (greater  than  1). 

MAXGP  out-of-range  (less  than  1). 

KRW  out-of-range  (greater  than  1). 

IOPT  out-of-range  (greater  than  1). 

KREJ  out-of  range  (greater  than  1). 

IRE5ID  out-of-range  (greater  than  3). 

IRUN  greater  than  l and  IRESID  greater  than  1. 

The  field  number  of  a predictor  card  is  greater  than  NV. 
The  field  number  of  a predictor  card  is  less  than  1. 

Either  KBL1  or  KBL2  out-of-range  (not  equal  to  zero 
or  1). 

Interval  length  out-of-range  (less  than  0). 

Number  of  categories  over  all  predictors  greater  than 
700. 

Number  of  categories  over  all  predictors  greater  than 

699. 

AID-4  predictor  card  (card  type  4)  expected. 

Continuation  card  incorrect. 

Number  of  categories  greater  than  40. 

Total  number  of  categories  greater  than  700. 

AID-4  criterion  card  (card  type  5)  expected. 

The  field  number  of  the  criterion  card  is  greater  than  NV. 
The  field  number  of  the  criterion  card  is  less  than  1. 

The  weight  field  in  the  criterion  card  is  greater  than  NV. 
The  weight  field  in  the  criterion  card  is  negative. 


YMAX  is  less  than  or  equal  to  YMIN. 

NV  ia  greater  than  83. 

Not  used. 

Interval  length  is  less  than  2ero. 

ICARD  greater  than  1. 

IRUN  greater  than  2 and  ICARD  equal  to  1. 
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An  Example  Problem: 

Figure  C-l  shows  the  input  de^k  for  a problem  with  12  data 
points.  The  first  four  cards  are  the  system  control  cards  required  to 
execute  AID4UT/AIDTRE.  There  are  three  predictor  variables,  (X^, 

X , X^).  Figure  C-2  shows  the  standard  output  set  produced  by  all 

A1D4UT  runs.  Figure  C-3  shows  the  detailed  output  for  the  first  itera- 
tion of  the  splitting  process.  This  detailed  output  is  optional  and  con- 
troled  by  the  input  parameter,  KSTOP. 

Figure  C-4  shows  the  residuals  table  for  groups  8 and  9 and 
the  summary  information  available  at  the  end  of  the  splitting  process. 

It  also  shows  the  BSS/TSST  and  BSS/TSS(i)  profiles  and  the  corresponding 
ranking  of  the  predictor  variables  based  on  the  PEP  coefficient  (RECSUM). 
This  latter  coefficient  was  proposed  by  Finifter  and  represents  the 
average  explanatory  power  of  a predictor  over  all  groups  in  the  AID- Tree. 
This  output  is  only  available  with  LRUN  < 1. 

Figure  C-5  presents  the  skeleton  version  of  the  AID- Tree 
that  is  produced  when  ITREE  - 2.  Figure  C-6  presents  the  detailed 
AID- Tree  that  is  produced  when  ITREE < 2.  If  an  AID- Tree  is  not 
desired  for  a run,  removal  of  the  EXECPF(4409,  AIDTRE)  card  will 
eliminate  the  tree  blit  will  not  affect  the  output  shown  in  Figures  C-2 
through  C-4. 


Setting  of  Parameters  in  AID4UT/ AIDTRE: 

In  initial  exploratory  analysis  runs  on  a data  set,  the  fol- 
lowing parameter  settings  are  recommended: 

IRUN  =0,  PI  = . 0001,  P2  = . 0001,  MAXGP  ~ 100,  NMIN  - 2, 

KSTOP  = 5,  IRESID  = 0,  ITREE  = 2. 

These  settings  allow  maximum  splitting  to  occur  and  detec- 
tion of  any  coding  errors  and  refinement  of  predictor  categories  (if 
these  categories  were  arbitrarily  defined  originally).  These  parameters 
do  allow  considerable  overfitting  to  occur  if  the  data  set  is  small  and 
further  runs  with  restricted  parameter  values  will  generally  be  required. 
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In  detailed  analysis  via-is,  the  critical  parameters  become 
PI,  P2,  MAXGP  and  NMIN.  These  values  are  dependent  upon  the  size 
of  the  data  set.  In  cases  where  there  are  less  than  500  data  units,  the 
following  values  are  suggested. 

PI  a.  01  P2S.005  MAXGP  £ 90 

NM1N  a 5 percent  of  data  set,  KSTOP  s 10 

Other  parameter  settings  are  dependent  only  on  the  output 
desires  of  the  analyst.  For  example,  the  residuals  list  may  only  be 
desired  on  final  runs. 


Use  of  AID4UT/ A1DTRE  with  other  Computer  Programs  as  a Policy 
Capturing  System: 


In  the  process  of  Policy  Capturing,  various  ancillary  com- 
puter programs  were  used  with  A1D4UT/ AIDTRE  to  analyze  and  develop 
regression  models.  These  programs  were: 


Program 

Purpose 

Documentation 

GCORR 

Contingency  Table  Generation 
for  Data  Analysis 

Anderberg  (1971) 

REG  REJ 
(EDSTAT-J) 

Multiple  regression  model 
determination  and  vross 
validation 

Jennings  (1971) 

MCA 

(SOCLIB) 

Multiple  Classification 
Analysis 

UT-  Computation 
Center 

STEP-01 

Stepwise  Regression 

CFHR,  ES3,  UT 
Austin 

NCRIT 

Prediction  of  Binary 
Vector 

(Listing  Included) 

Figure  C-7  is  a Process  Diagram  for  using  the  various 
programs  in  the  Policy  Capturing  process.  Alternative  coefficient 
determination  schemes  allow  the  analyst  considerable  flexibility.  The 


PROCESS  DIAGRAM 
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use  of  the  DD^TAT-j  gystem.  represents  a particularly  important  option 
since  the  analyst  can  write  a driver  program  which  uses  the  standard 
statistical  subroutines  to  perform  a statistical  analysis  that  is  tailored 
to  his  specific  needs.  As  an  example,  the  listing  for  a driver  program 
named  CRV  is  given  later  in  this  appendix.  This  is  the  routine  that  was 
written  to  perform  the  interative  deter mination/ cross- validation  of  re- 
gression coefficients  discussed  in  Chapter  IV.  A complete  description 
of  the  EDSTAT-J  package  is  given  in  Jennings  (1971). 

In  the  cross-validation  phase  of  the  Policy  Capturing  process, 
either  the  PREDICT  routine  of  the  EDSTAT-J  package  or  the  NCRIT 
routine  can  be  used.  The  NCRIT  routine  was  adapted  for  use  in  this 
research  from  a program  obtained  from  the  AFHRL.  It  is  very  similar 
to  the  PREDICT  routine.  It  does  allow  the  specification  of  an  arbitrary 
objective  function  and  operates  as  a "canned  program".  A complete 
listing  of  NCR  IT,  including  comment  cards  describing  the  input,  is 
included. 


Program  Listings: 


In  order  that  a master  listing  of  the  programs  used  in  this 
research  could  be  referenced,  source  listings  for  those  programs  that 
are  not  formally  described  elsewhere  are  provided.  The  programs 
included  are: 


AID4UT  (AIDIV,  ERROR,  HEADER,  LINK3,  AID  FIN,  RANKSIT) 

AIDTRE  (TREEC.  BLOCK,  PLEVEL,  PLOTH,  PLOTH1 , PLOTV, 

PLOTV! , STOAl , PLOTHI,  ITOA1 , PLOTHF,  RTOA1 , AIDPLT) 

CRV  (The  various  subroutines  referenced  are  described  in 

Jennings,  1971) 

NCRIT  f DRIVER,  DA  TRAM,  TMT,  R.EGWTS,  TABLES,  OBJFUN) 

( This  program  also  uses  some  EDSTAT-J  routines.  ) 
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